This paper describes a scene analysis system which can perform semantic segmentation of a scene with hierarchical structure. An input color image is first segmented into a structured symbolic description using intensity data. Semantic interpretation is performed on this description. Top-down control and bottom-up control are combined in the framework of semantic region growing. Knowledge is represented as a semantic network in the bottom-up process and in the top-down process as a set of production rules. The system was applied to outdoor scenes.