I don't think I entered an art gallery until I went to Europe as a backpacker, by which time I had enough education to understand some of the historical and cultural context of what I was seeing.Yet you did not gain enough education to realise when you have sabotaged your own argument. You did not visit an art gallery until you were an adult, yet you want parents to stop taking their children to art galleries. Or, to put it another way, you missed out on a huge part of childhood education and experience, and so should today's children. Surely, if art galleries are a good thing, then children should experience them.
The problem with you argument is that art is not just historical and cultural context. That sort of thing can be found on the labels, in the books and in the Art History departments. The real thing about art is the experience of it. To make an analogy, one can know about architecture - one could read Pevsner's An Outline of European Architecture and similar books thoroughly, taking notes and committing images to memory. But that is not the same as experiencing the buildings. Art needs to be experienced - reproductions and explanations in books are not the same as the experiences of the objects.
The funny thing about your argument - apart from the assertion that a backpacker's helter-skelter experience of art galleries is superior to a childhood of growing up with art - is that you make it in writing. Now, presumably you did not learn to read on your OE. Presumably, you were by then a capable reader. Probably, you were taught to read by direct experience of writing (unless you learned by synthetic phonics, in which case I pity you and suggest you read Michael Rosen on the subject; better still, watch this short film he has made - it won't take long). My point is that the only way to become a writer is to be a reader. The same is true of art. Unless children see art they will never understand it or do it.
Children deserve early experiences of art. Besides, if they don't grow up with art they might grow up feeling intimidated by art galleries or thinking they are places one only visits on backpacking holidays. They might also grow up with historicist views of art - that it was done in the past, rather than something that is actual, something that continues to happen, something they can do.
Keeping children away from art until they are old enough to know the context also risks them thinking of art as no more than illustration - that, say, the value of a paining by Hogarth is in showing what life was like in the 18th century and not much else. They might also grow up to judge art works by the supposed quality of depiction - that a Hogarth is necessarily better than a Hodgkin because you can see what is going on. Worse still, they might think that non-representational art is not art, that a child could do it. They might also think that representational art is is realistic, that depictions are how things really are. This can lead to an awful lot of trouble.
Besides, what harm does it do? Some children are allowed to run wild by their ghastly middle-class parents who have just bumped into their ghastly middle-class friends and simply must tell all about how wonderfully those children are doing at their marvellous schools, while ignoring them. But the problem there, as always, is the parents. Rather than excluding the neglected offspring, it might be better to remove the parents to some place where they can natter inanely but harmlessly, such as the ubiquitous coffee shop or the inevitable atrium. The innocents could then be guided through the galleries by empathetic staff, who would show them everything their parents are missing because of their futile self-obsession and ambition. The children then might have the opportunity to grow up bohemian, breaking the cycle of middle-class ghastliness and raising the aesthetic standards of our declining civilisation.
Surely, this can only be a good thing.