Designed in 1886 by local architect Alan Crombie, this massive building was commissioned for the production of Scottish tweed and became one of several tweed mills in the city. 

Since closing its doors to textile production in the 1920s, the building has housed many industries, but the most notable came when Germany invaded and occupied Norway in 1940. The soldiers, sailors, and other patriots who escaped from the country to continue the struggle alongside Britain were allocated the building as the headquarters of the Norwegian forces in the United Kingdom. Over 1,000 Norwegians were billeted in Rosefield Mills, before a larger barracks complex was built at Carronbridge, which lies north of the town.

Amongst the earliest arrivals were about 250 members of the Norwegian whaling fleet who were at sea when Germany invaded and refused to return to Norway. In 2019, to commemorate their arrival in Dumfries, a whale-shaped bench was installed in Dock Park directly opposite the mill building. 

In 2019, the building was purchased by the Dumfries Historic Buildings Trust, but still sits crumbling while awaiting a new life.