hckrnws
Have a damaged painting? Restore it in just hours with an AI-generated “mask”
by WithinReason
[delayed]
I will reserve judgement on this for the simple fact is that att conservation has been responsible for a huge amount of Art vandalism.
The 'restoration' of the cistine chapel ceiling was funded by a Japanese tv company. The cheapest approach was chosen which assumed that michelangelo made absolutely no corrections to his fresco using applied paint. It is perfectly obvious that this was a mistaken assumption, in the process removing many of the artists original work. I can upload some slides later if anyone is interested.
In victorian times many classic sculptures were scrubbed of their original paint and their stonework bleached, just in order to serve the tastes of the time.
And let us not forget that modern conservators will add or remove elements according to the clients taste. Eg change the flag of a ship from British to American.
In that case they explicitly says that the process is non destructive and reversible. Essentially a wrap over the original artwork
> The restoration is printed on a very thin polymer film, in the form of a mask that can be aligned and adhered to an original painting. It can also be easily removed. Kachkine says that a digital file of the mask can be stored and referred to by future conservators, to see exactly what changes were made to restore the original painting.
I wonder whether it's still worthwhile to replaced a yellowed varnish varnish on old painting, just to be sure that it doesn't degrade further (with the assumption that historical varnish are somewhat lesser than modern one, which I really don't know if it's the case).
> I wonder whether it's still worthwhile to replaced a yellowed varnish varnish on old painting, just to be sure that it doesn't degrade further (with the assumption that historical varnish are somewhat lesser than modern one, which I really don't know if it's the case).
I would say that we should just live with the difference. There is a near guarantee that our experience of paintings as they exist today will always be significantly different to how they would have been experienced when they had been made, regardless of what we do. The factors are:
- Yellowing of oil/pigment/varnish. This is pretty much inevitable, even in modern paints.
- Fugitive colors. Most high-color pigments tended to fade significantly over time. Degas was once seen crying in front of a Delacroix, upset at how badly the colors had faded, even in his lifetime (It might have been Manet... can't locate the relevant anecdote online).
- Excessive varnishing. It used to be so that paintings received regular varnishing by their owners. Over time, the layers of varnish would build up to a ridiculous degree.
- Lighting. For me this is probably the most annoying factor. A general rule is that paintings should be lighted in the same light that they were painted. The values go to hell if a painting is lighted too brightly. There are many so-called hanging experts, who always light too strongly, even in respected national collections.
Semi-related anecdote: The first time I went to the Rodin museum I was struck by how badly one of the busts had been lighted. Whilst the guards were not looking, I moved it to a more agreeable position. I came back one year later to find it was still in that position.
As for using modern varnishes... I am not expert, but AFAIK one factor that accounts for the amazing longevity of old paintings was the compatibility between their many layers. The canvas, the primer, the paint and the medium were all derived from the same plant: flax. On top of that would sometimes be laid stand oil, again derived from flax. If this continuity is broken, all manner of problems might arise, most of which would be long term issues such as cracking and flaking.
> AFAIK one factor that accounts for the amazing longevity of old paintings was the compatibility between their many layers. The canvas, the primer, the paint and the medium were all derived from the same plant: flax.
AFAIK that's largely untrue in the general case:
- Canvas painting only appeared in the 15th century, and took a while to spread, paintings before the 17th century (16th in some locations) tend to be on wood panels e.g. raphael and da vinci painted almost exclusively on board.
- Canvas is almost always coated with gesso which at the time would use rabbitskin glue as binder, no flax there.
- While flaxseed was the most common drying oil (in europe), walnut, poppyseed, and safflower, were also in use. And additives were usually mixed in to manage the viscosity of the paint, so even in the "best" case it's not like the paint would be just pigments in linseed oil unless it's a very early oil painting, which wouldn't have been on canvas.
- As for varnishes, not only was flax not the only drying oil for oil varnishes, varnishes could also be "spirit" varnishes (with a resin dissolved into a solvent like alcohol or turpentine), waxes were also sometimes used as or in varnishes.
True… what I said only applies to canvas paintings. I should have made that clear.
The longevity of murals is easy to account for: the paint is applied to wet plaster, in that way becoming part of the wall. That is why the murals of Pompeii survived.
> Canvas is almost always coated with gesso which at the time would use rabbitskin glue as binder, no flax there.
You are right about the rabbit skin glue, but wrong about the gesso. As I recal, traditional Gesso is a mix of glue plus titanium white powder and is very brittle, generally unsuited to a flexible support such as canvas. A canvas painter would more likely use something flexible like a mix of pigment and rabbit skin glue, or pigment and egg protein or pigment and oil.
I thank god for modern primers. Using modern primers, I can prime a canvas in two days. An oil based oil primer could take months to dry.
As far as I can tell, the ingredients described by Cennini[1] is close to the “traditional” preparation:
* Gypsum (Hydrated calcium sulfate)
* Zinc white pigment
* Clean tap water or distilled water
* Rabbit skin glue
—-
[1] Cennini, Cennino d'Andrea, The Craftsman's Handbook "Il Libro dell Arte," Daniel V. Thompson, Jr., trans. (New York: Dover Publications, 1960) pp. 69–74.
> Semi-related anecdote: The first time I went to the Rodin museum I was struck by how badly one of the busts had been lighted. Whilst the guards were not looking, I moved it to a more agreeable position. I came back one year later to find it was still in that position.
That made me think of the old German sketch "das Bild haengt shief" (the picture is askew): https://www.youtube.com/watch?v=a6WQaIIZ248
I hope you left the rest of the exposition standing.
I would assume the mask only covers the damaged areas. This means it replaces manual retouching, so you still need to clean the original painting and remove the varnish in order to get at the original colors, and as part of the usual cleanups (removal of old conservation, stabilisation, infill). The article specifically says that the film is adhered to and via a varnish layer.
Makes me wonder how it would handle heavy impasto tho.
Obviously it can't handle heavy impasto. With heavy impasto, people usually literally sculp the surface using some material to math the rest of the painting and then paint on top of this. Even with minimal impasto, they will often try to mimic the texture by imprinting various tools into the material, mimicking for example brushes used by the original author. This printing method can't do this at all, and the filter might struggle when the painting is too uneven.
Otherwise this might be an interesting technique, if the result can match the color and texture of paint perfectly. I can see it being used for some low priority paintings. There are much more paintinga that need restoration than people with necessary skills, so this could save of them, as it will be more viable to fix them. The infilling is usually just a small part of the entire process, but usually the most difficult wrt how skillful the conservator must be. You must be able to match colors and style perfectly, and there are huge differences in how fast this process is depending on the skills of the painter.
Comment was deleted :(
And don’t forget the restoration attempt on the Ecce Homo portrait[0]
[0] https://en.wikipedia.org/wiki/Ecce_Homo_(Garc%C3%ADa_Mart%C3...
Honestly, that's a lot better than the original. We should restore more art that way.
> I can upload some slides later if anyone is interested.
I am!
The restoration method is more interesting than the AI, but claiming it uses AI is probably necessary to get funding for it.
Yes, it seems like the same method (a thin polymer mask of which a computer record exists) could be used in a completely human-driven process. Just allowing restorers to try a few different things to see what works would probably speed up their process immensely.
There's probably some other grant money they can get down the line for a completely AI-less process, but obviously there's a lot more funding for AI than for art restoration.
It's conceptually similar to current restoration techniques, but, the real innovation is in doing the whole thing as one piece ahead of time rather than requiring somebody to get in there with tiny brushes after a transparent base layer is completed.
It's also worth noting that this doesn't help with the first half of most restorations, which is removing crud and wear and often painstakingly undoing previous alterations or low-quality restorations. There's actually a pretty long history of paintings being altered to fit current fashions or otherwise mucked with in ways that make modern art historians cringe.
As far as I understand this only replaces the retouching phase. A human needs to do that in place in order to color and pattern match (and it's already hard enough that way). And I do not know but would assume the retouching paints can be removed without affecting the isolation layer if the conservator realises they went in the wrong direction earlier and can’t easily recover.
A few years ago, they would just call it an "algorithm" or simply "software"
They could probably just tell people they're "using AI" and then just use whatever methods work best. It's not like investors are doing much more DD than listening for that phrase.
I would’ve argued against this so hard if it weren’t for that NIKOLA company advertising their trucks running on an “HTML supercomputer” and still getting millions in funding.
Funny aside, they have an odd ligature to for “fi” on that site. It shows up in iOS Safari at least.
> Kachkine acknowledges that, as with any restoration project, there are ethical issues to consider, in terms of whether a restored version is an appropriate representation of an artist’s original style and intent. Any application of his new method, he says, should be done in consultation with conservators with knowledge of a painting’s history and origins.
A sort of interesting thought, “what was the artist’s intent, should be recover the painting,” is a well known question nowadays. It would be sort of funny if current artists would just write down what sort of restoration plans they are ok with. I wonder how many would say “just do what you will do people can enjoy it.”
Although, one could argue maybe that the damage which occurs to artwork as it ages also tells a historical story. Perhaps that story doesn’t just belong to the artist, and so restoring the work could be questionable even with their permission. I’m sure this is well-trod ground.
Unless the result is a form of protection to the painting then why not "just" reprint the whole painting and not touch the original at all?
Looking at a oil painting in real life is a vastly different experience than a looking at a print. Oil paintings are for a start 3D (textured). Colour reproduction is also difficult.
It changes the medium. Even if it works perfectly, doesn't alter luminosity and perfectly preserves the art under the wrap, it's still shrink wrapped art and alters the experience of viewing the work.
Perhaps.. have you seen the 3d replica's in the van Gogh museum? The film could also have a 3d printed version.
I doubt this technique could be used with heavy impasto paintings. Wouldn't the film have to sit on a mostly flat image to work correctly? Otherwise it would cast shadows between the painting and the film.
3d printing impasto does sound cool as a tool for restoration of topographically scanned originals.
Infill the damaged parts of this painting with exactly no elephants.
Why not just generate a new painting with AI? How much of the painting has to be covered with a mask before it’s not a painting anymore?
That’s not exactly a gotcha. Art conservators have been grappling with variants of that question for hundreds of years.
The answer might just be: What's done is done. Conserving what's left is the best you can hope for.
That is a bit of a truism, as conservators don't have access to time magic and can only conserve what's left. The question is how far conservation extends.
To my layman's understanding, the current logic of conservation is:
1. clean. dust, grime, soot, tar, and other environmental deposits don't belong on the painting and have little value
2. stabilise, this is mostly for paintings which are actively degrading (e.g. paint is lifting), this can also include repairs to the substrate e.g. patch up tears in the canvas
3. fill-in losses
The last one is the one that has some contention, and what TFA is about (and only a portion of it to boot), and the idea behind it is (again to my understanding) that at the end of the day art is for the living to appreciate, and while damage can be of some historical interest it generally detracts from enjoyment of the piece and is (usually) not part of the original vision.
However it is of course subject to (0). do not damage what remains, so the retouching should be well separated from the original material in order to be identifiable and removable without risk of further damage in case re-conservation is needed, or better conservation methods can be used.
And then if you allow for (3) arises the question of limit, is there and if so what is the point at which it doesn't make artistic sense to try and replace losses even if somebody's paying for it, and it is better to accept the piece's new state of being as its normal?
> "Because there's a digital record of what mask was used, in 100 years, the next time someone is working with this, they'll have an extremely clear understanding of what was done to the painting"
Reminds me of https://xkcd.com/1683/
Whatever digital record format they use, they will probably need to rewrite it once every few years to ensure that it's still readable.
but most of the lift looks mechanical : masking, polymer filling, pigment blending. if ai's involved, it's probably color inference or mask prediction. nothing wrong with that, just feels overstated. curious if they benchmarked against expert restorers on visual fidelity. and who signs off on historical accuracy here? ai or the human behind the brush?
No thank you, that would destroy the value and art of the painting.
Comment was deleted :(
I hope the AI will not generate hands with seven fingers ...
That was a problem a year or two ago. New AI models have gotten past that point now.
openai native 4o image generation generates 4 fingers instead.
Is that a joke? Because 4o image generation (assuming you click "Image Generate" which uses gpt-image-1) EASILY handles rendering hands with the proper number of fingers even without specifying something like that.
If anything, it's actually MORE difficult to generate hands with an improper number of fingers. Apologies to Count Rugen.
The tweet about gpt 4o image generation capabilities release had 4 fingers.
I'd like to see that tweet if you have it. I also don't remember the intro page they put out having any such issue:
Look at the hands in "selfie view of the photographer, as she turns around to high five him".
Hmm, I agree it's not a good high-five - their hands aren't really centered against each other, but her raised right-hand is oriented at an angle where her thumb and index finger are kind of obscured since her pinky finger is closest to the camera.
I thought the man's hand was either missing a pinky finger or had an abnormally short one.
Here's another one, this tweeted image shows either six fingers or an abnormally large middle finger: https://x.com/OpenAI/status/1904602845221187829
Thanks for that one - yeah that one is definitely NOT great. That middle finger would have to be admiral ackbar sized to pass for a single digit.
Reminds me of when Karpathy tweeted about the founding of an educational AI lab:
If you consider finger count to be the lowest bar, then it's still a problem. People just aren't generally creative enough to find the cracks.
Think in terms of hands, their components, and their function, and test again. Be specific.
So? My point wasn't that they were capable of EVERYTHING - it was addressing what looked like to me (and likely to any casual observer) factually incorrect information.
You're also really talking to the wrong person about potential deficiencies in GenAI for images.
I run an entire site where I compare a multitude of prompts I CREATED to explicitly test state of the art major generative image providers (Imagen4, gpt-image-1, Flux Kontext, etc.) - I'm all too aware of their shortcomings.
Didn't mean to wind you up! Totally wasn't my intent. Looking over your site, I feel like my point is pretty strongly reflected by your work, though.
While models have been trained to deliver high-level impressions (with increasing attention to detailed problem domains), one-shot control is still relatively poor, and they lack the fundamental skill of a trained artist. There are chasms between what you think you're prompting, what the text encoder understands, and how the model interprets that input, with the resulting effect of a professional musician intentionally playing badly... hands not excepted.
For instance, in "Mermaid Disciplinary Committee" on your site, every hand has a deformity or finger count inconsistency. In "Spheron", the hands have no variation and suffer from cross-subject cloning (even 4o - look at the shield-carrier).
That's what I meant about creativity and being specific. Try prompting for three people holding up certain fingers on one or both hands. Start with the index, progress to pinky. Ask it to show you a hand gripping things, rotated in different orientations. Prompt for a hand with 3 fingers, then 6 fingers, then no fingers. Ask for gang signs or shadow puppets, pinching something, with fewer or extra digits. The illusion breaks down quickly.
This is a space I'm working in, retraining text encoders and diffusion models to understand the same things first year arts students learn. With how limited and poisoned most models are, it's been a huge effort.
That's a really cool website. You should consider submitting it for show HN, I'm sure others would enjoy it as well.
I submitted it (https://news.ycombinator.com/item?id=44281555) as I have found it interesting, too. If author wants me to (in time), I will delete it so he can re-submit.
And a thumb? Or like The Simpsons?
Where can I read the theorem that guarantees this?
It’s a perfect example for thinking about the limitations of these things. Locally fingers appear next to other fingers, with a high probability. It requires a level operation to count how many fingers exist on a component and understand their relative position.
tell me you didn’t even glance at the article without telling me you didn’t even glance at the article
No links to Baumgartner from HN yet? Strange.
For those who are unaware, as I was, this is an art restoration business and YouTube channel:
Crafted by Rajat
Source Code