fix: add X-Robots-Tag header to prevent robot indexing and following

[itslenny]

What kind of change does this PR introduce?

Bug fix

What is the current behavior?

Many objects are indexed by web crawlers as there are no x-robot-tag headers

What is the new behavior?

• Add default x-robots-tag: none header to prevent search engines from indexing / following
• Add ability to specify custom x-robots-tag header when uploading objects
• Use x-robots-tag value specified in object metadata in headers when serving files
• Fix eslint warnings in edited files

Additional context

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Robots-Tag

[coveralls]

Pull Request Test Coverage Report for Build 19315179516

Details

• 174 of 245 (71.02%) changed or added relevant lines in 13 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.04%) to 77.743%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/http/routes/object/getPublicObject.ts	1	3	33.33%
src/http/routes/object/getObject.ts	3	7	42.86%
src/storage/validators/x-robots-tag.ts	123	188	65.43%

Totals
Change from base Build 19309174116:	-0.04%
Covered Lines:	24773
Relevant Lines:	31542

---

💛 - Coveralls

[ferhatelmas]

Ignore me if this is already discussed but otherwise wanted let you know that nocache/noarchive seem still in use by some (i.e. Bing) and so excluding them can fail normally valid requests in validation