Kelly's oil paintings go to the heart of abstract art's challenge precisely because they are, in plain fact, nice to look at. White Curve (1974) creates an amazing lift and lightness, its white, bird-like shape ready to soar up into the heights of the gallery. Why not stop and enjoy it? Perhaps it is "chromophobia", fear of colour, which surprisingly enough has something to do with Islamophobia.

Abstract art is an Islamic invention. The first art to be entirely without story or reference was created in Islamic villas and palaces and adapted from Roman mosaics; in the middle ages, this non-figurative system of decoration was taken to new heights by a culture in love with mathematics. Medieval Muslim mathematicians mastered and improved ancient Greek geometry. European - and American - art has by contrast always been about meaning: it tells stories, depicts reality. The art historian EH Gombrich, who hated abstract art, thought it was a decadent assimilation of non-western decorative prettiness, the end of the western tradition of representation. But had not reckoned on video art. It seems to me that video is simply the current form taken by western art's ancient predilection for narrative and reality. Another new gallery at Tate Modern shows a Douglas Gordon film that slows down archive footage of a shell-shocked soldier. It is disturbing and moving. What in comparison do you get from Kelly except pretty colours?

The genius of Kelly lies precisely in his refusal to give, or rather claim, more. When he began his career in the late 1940s, American art was besotted with abstraction - but abstraction that claimed to be about deep emotion. Mark Rothko and Barnett Newman proclaimed the "sublimity" of their art, and whatever you make of Tate Modern's Rothko room you can't miss its sense of moral purpose. Kelly ignored abstract expressionism. Just when New York - in the words of a French Marxist - was "stealing the idea of modern art" from Paris , he settled in Paris . It was while living in France from 1948 to 1954 that he discovered his own personal version of abstract art, and you actually see him making this discovery in the most important painting in the new display.

Méditerannéeis one of a series of oil paintings Kelly made in the early 1950s that shuffle colours. Exactly contemporary with abstract expressionism, it emphatically rejects that style's drama. Kelly has simply painted nine wooden panels in nine different colours: each colour is applied evenly and smoothly, as if he were painting a piece of furniture. Then he has arranged the panels to make one large rectangle, with an uneven surface. He includes the three traditional primary colours - blue, yellow, red, adjacent to each other around the painting's top right corner. They act as flaming counterpoints to a flow of darkness: an L-shape of blues, a shiny black that a creamy white separates from a duller black.

The colours of sea, rocks, beach and sun ... a Mediterranean seascape. Yet there is something so structured, so intelligent, about the way this information reaches you. What Kelly is doing is to break up light into separate colours much as the pointillist or divisionist painters did in late 19th-century France. Look at an oil painting by Seurat and it is all coloured dots: Kelly has massively increased the scale of those dots, and turned them into squares. At the same time this is a decorative art, as mathematical, as unemotional as the tiles in the Alhambra When you look at Méditerannée you wonder: did Kelly arrange these colours to suggest the sea and its painters, or did he move them randomly and then notice a Mediterranean quality to those colours?

There is nothing so deliciously melancholy as really rich colours in art: the very joy of colour, that can transport us to paradise, unlocks our deepest longings, and we soon realise paradise is not on this earth. Kelly's art is at once upbeat and sad, transporting and introspective, like the jazz of the 1950s, like Miles Davis. It is all of that without ever telling a story, pointing to a moral. In this it has a masterly precedent.

I said you can follow the history of abstract art at Tate Modern, and so you can. There is a work in the Tate collection, too great to ever go off display, that closely resembles Kelly's method when he made Méditerannée. When this young American artist was arranging colours in France in the 1950s, so was a French titan of modern art. In 1953, Henri Matisse created Tate Modern's The Snail, one of his late masterpieces constructed from cut-up painted papers. Those blocks of pure colour, arranged in his last years, are as uninhibited as Kelly's in treating the eye to pure optical pleasure that breeds nostalgia, memory, mystery.

Tate Modern's new Kelly display is in its Level 5 suite entitled Idea and Object: Around Minimalism. Kelly was an anomaly in 1950s American art, but he suddenly became a star in the 1960s because his cool approach to making abstract art seemed to herald the new style critics called "minimalist". In Tate Modern you can see how Kelly's art relates to the 1960s sculpture of Donald Judd, Carl Andre and Dan Flavin. The way Kelly arranges his wooden blocks of colour in Méditerannée anticipates Andre's arrangements of metal plates on the ground. Most of all, he makes you see how minimalism is an art of colour. Flavin's fluorescent lights create rich and emotional displays of colour that are far more baroque than Kelly's reserved pictures. Judd's sculptures frame and contain colour; like alchemists' jars, they hold blocks of blue and red as if these were rare and precious magical substances. This is a man who worships colour so much he has to store it in an industrially fabricated reliquary.

If Kelly makes you see the sheer beauty of minimalism - as opposed to the ready-made conceptualism it is so often seen as a dumb vessel of - he also connects contemporary, living art with the heritage of Matisse. This makes him one of the most important artists alive, and Tate Modern should maintain this beautiful display of his work as a permanent exhibit. Could a child have made these paintings? More to the point is that, standing in front of Kelly's Broadway, as colour moves towards me in a fiery surge, I feel like a newborn child, seeing the world for the first time. Because abstract can't be explained away, it can't be exhausted. It is always new.

Ellsworth Kellyis one of 21 new displays in the UBS Openings: Tate Modern Collection, London , which opens on May 4.