ML agents (64,64,3) vs [3,64,64] error | Official Unity Discord | Page 1

bold magnet Sep 16, 2025, 10:40 AM

#

for multiple hours i tried to fix it, but nothing works. even custom wrapper is not possible due to protection. i am hopeless and i relly need it to work or idk what i would do. I am using ver very simple input from camera sensor yet after running the training it keeps throwing this error:
mlagents_envs.exception.UnityObservationException: Decompressed observation did not have the expected shape - decompressed had (64, 64, 3) but expected [3, 64, 64]
it is supposed to be automatically converted but no

true rock Sep 22, 2025, 11:42 PM

#

bold magnet for multiple hours i tried to fix it, but nothing works. even custom wrapper is ...

Hey - same here. I literally joined the Unity Discord just to ask about this one error and see if I was doing something obviously wrong, but this behavior appears to be straight out of the box. I've used Unity ML-Agents for about 4 years, but now with Unity 6, using the built-in camera observation without modifying rpc_utils.py results in this shape error. (The only real alternative is to use Unity < 6.0 and its associated ML-Agents Python Packages.) I have no idea why this change was even necessary - or at least why the rest of the package didn't conform?

The issue itself is that the visual observation should be in (Channels, Height, Width) (C, H, W) but the sensor is providing (H, W, C). Ok - right, that's a common issue; people often arbitrarily pick whether the channels should be the first or last dimension. What's puzzling is how low-level this issue but it hasn't gotten resolved?

I've fixed this myself by opening Lib/site-packages/mlagents_envs/rpc_utils.py (You'll need to know where your Anaconda/minconda env/Python environment is first.)

Make the change below:

#


def behavior_spec_from_proto(
    brain_param_proto: BrainParametersProto, agent_info: AgentInfoProto
) -> BehaviorSpec:
    """
    Converts brain parameter and agent info proto to BehaviorSpec object.
    :param brain_param_proto: protobuf object.
    :param agent_info: protobuf object.
    :return: BehaviorSpec object.
    """
    observation_specs = []
    for obs in agent_info.observations:
        # Heuristic: some environments (or Unity versions) may report visual shapes as CHW [C,H,W]
        # while the data is actually sent as HWC [H,W,C]. Try to detect and correct this early.
        reported_shape = tuple(obs.shape)
        corrected_shape = reported_shape
        is_visual_chw = (
            len(reported_shape) == 3
            and reported_shape[0] in (1, 3, 4, 6, 9, 12)
            and reported_shape[2] >= 16
            and reported_shape[1] == reported_shape[2]
        )
        if is_visual_chw:
            # Looks like CHW; convert to HWC
            corrected_shape = (reported_shape[1], reported_shape[2], reported_shape[0])
            try:
                warnings.warn(
                    f"Observation shape reported as {reported_shape}; correcting to {corrected_shape} (assuming CHW->HWC).",
                    RuntimeWarning,
                )
            except NameError:
                # warnings may not be imported in older versions; ignore
                pass
        # Handle dimension properties: if provided and we corrected shape, rotate properties accordingly
        if len(obs.dimension_properties) > 0:
            if is_visual_chw:
                # Input assumed (C,H,W). Output should be (H,W,C). Rotate properties to match.
                dim_props = tuple(DimensionProperty(dim) for dim in obs.dimension_properties)
                if len(dim_props) == 3:
                    # Move first elem (channels) to last
                    corrected_dim_props = (dim_props[1], dim_props[2], dim_props[0])
                else:
                    corrected_dim_props = dim_props
            else:
                corrected_dim_props = tuple(
                    DimensionProperty(dim) for dim in obs.dimension_properties
                )
        else:
            corrected_dim_props = (DimensionProperty.UNSPECIFIED,) * len(corrected_shape)
        observation_specs.append(
            ObservationSpec(
                name=obs.name,
                shape=tuple(corrected_shape),
                observation_type=ObservationType(obs.observation_type),
                dimension_property=tuple(corrected_dim_props),
            )
        )

    # proto from communicator < v1.3 does not set action spec, use deprecated fields instead
    if (
        brain_param_proto.action_spec.num_continuous_actions == 0
        and brain_param_proto.action_spec.num_discrete_actions == 0
    ):
        if brain_param_proto.vector_action_space_type_deprecated == 1:
            action_spec = ActionSpec(
                brain_param_proto.vector_action_size_deprecated[0], ()
            )
        else:
            action_spec = ActionSpec(
                0, tuple(brain_param_proto.vector_action_size_deprecated)
            )
    else:
        action_spec_proto = brain_param_proto.action_spec
        action_spec = ActionSpec(
            action_spec_proto.num_continuous_actions,
            tuple(branch for branch in action_spec_proto.discrete_branch_sizes),
        )
    return BehaviorSpec(observation_specs, action_spec)