perf(federation): Ed25519署名に対応する (#13464)

* 1. ed25519キーペアを発行・Personとして公開鍵を送受信

* validate additionalPublicKeys

* getAuthUserFromApIdはmainを選ぶ

* :v:

* fix

* signatureAlgorithm

* set publicKeyCache lifetime

* refresh

* httpMessageSignatureAcceptable

* ED25519_SIGNED_ALGORITHM

* ED25519_PUBLIC_KEY_SIGNATURE_ALGORITHM

* remove sign additionalPublicKeys signature requirements

* httpMessageSignaturesSupported

* httpMessageSignaturesImplementationLevel

* httpMessageSignaturesImplementationLevel: '01'

* perf(federation): Use hint for getAuthUserFromApId (#13470)

* Hint for getAuthUserFromApId

* とどのつまりこれでいいのか？

* use @misskey-dev/node-http-message-signatures

* fix

* signedPost, signedGet

* ap-request.tsを復活させる

* remove digest prerender

* fix test?

* fix test

* add httpMessageSignaturesImplementationLevel to FederationInstance

* ManyToOne

* fetchPersonWithRenewal

* exactKey

* :v:

* use const

* use gen-key-pair fn. from  '@misskey-dev/node-http-message-signatures'

* update node-http-message-signatures

* fix

* @misskey-dev/node-http-message-signatures@0.0.0-alpha.11

* getAuthUserFromApIdでupdatePersonの頻度を増やす

* cacheRaw.date

* use requiredInputs
https://github.com/misskey-dev/misskey/pull/13464#discussion_r1509964359

* update @misskey-dev/node-http-message-signatures

* clean up

* err msg

* fix(backend): fetchInstanceMetadataのLockが永遠に解除されない問題を修正

Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com>

* fix httpMessageSignaturesImplementationLevel validation

* fix test

* fix

* comment

* comment

* improve test

* fix

* use Promise.all in genRSAAndEd25519KeyPair

* refreshAndprepareEd25519KeyPair

* refreshAndfindKey

* commetn

* refactor public keys add

* digestプリレンダを復活させる

RFC実装時にどうするか考える

* fix, async

* fix

* !== true

* use save

* Deliver update person when new key generated (not tested)
https://github.com/misskey-dev/misskey/pull/13464#issuecomment-1977049061

* 循環参照で落ちるのを解消？

* fix?

* Revert "fix?"

This reverts commit 0082f6f8e8c5d5febd14933ba9a1ac643f70ca92.

* a

* logger

* log

* change logger

* 秘密鍵の変更は、フラグではなく鍵を引き回すようにする

* addAllKnowingSharedInboxRecipe

* nanka meccha kaeta

* delivre

* キャッシュ有効チェックはロック取得前に行う

* @misskey-dev/node-http-message-signatures@0.0.3

* PrivateKeyPem

* getLocalUserPrivateKey

* fix test

* if

* fix ap-request

* update node-http-message-signatures

* fix type error

* update package

* fix type

* update package

* retry no key

* @misskey-dev/node-http-message-signatures@0.0.8

* fix type error

* log keyid

* logger

* db-resolver

* JSON.stringify

* HTTP Signatureがなかったり使えなかったりしそうな場合にLD Signatureを活用するように

* inbox-delayed use actor if no signature

* ユーザーとキーの同一性チェックはhostの一致にする

* log signature parse err

* save array

* とりあえずtryで囲っておく

* fetchPersonWithRenewalでエラーが起きたら古いデータを返す

* use transactionalEntityManager

* fix spdx

* @misskey-dev/node-http-message-signatures@0.0.10

* add comment

* fix

* publicKeyに配列が入ってもいいようにする
https://github.com/misskey-dev/misskey/pull/13950

* define additionalPublicKeys

* fix

* merge fix

* refreshAndprepareEd25519KeyPair → refreshAndPrepareEd25519KeyPair

* remove gen-key-pair.ts

* defaultMaxListeners = 512

* Revert "defaultMaxListeners = 512"

This reverts commit f2c412c18057a9300540794ccbe4dfbf6d259ed6.

* genRSAAndEd25519KeyPairではキーを直列に生成する?

* maxConcurrency: 8

* maxConcurrency: 16

* maxConcurrency: 8

* Revert "genRSAAndEd25519KeyPairではキーを直列に生成する?"

This reverts commit d0aada55c1ed5aa98f18731ec82f3ac5eb5a6c16.

* maxWorkers: '90%'

* Revert "maxWorkers: '90%'"

This reverts commit 9e0a93f110456320d6485a871f014f7cdab29b33.

* e2e/timelines.tsで個々のテストに対するtimeoutを削除, maxConcurrency: 32

* better error handling of this.userPublickeysRepository.delete

* better comment

* set result to keypairEntityCache

* deliverJobConcurrency: 16, deliverJobPerSec: 1024, inboxJobConcurrency: 4

* inboxJobPerSec: 64

* delete request.headers['host'];

* fix

* // node-fetch will generate this for us. if we keep 'Host', it won't change with redirects!

* move delete host

* modify comment

* modify comment

* fix correct → collect

* refreshAndfindKey → refreshAndFindKey

* modify comment

* modify attachLdSignature

* getApId, InboxProcessorService

* TODO

* [skip ci] add CHANGELOG

---------

Co-authored-by: MeiMei <30769358+mei23@users.noreply.github.com>
Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com>

1 files changed, 40 insertions, 21 deletions

diff --git a/packages/backend/src/core/FetchInstanceMetadataService.ts b/packages/backend/src/core/FetchInstanceMetadataService.ts
index aa16468ecb..dc53c8711d 100644
--- a/packages/backend/src/core/FetchInstanceMetadataService.ts
+++ b/packages/backend/src/core/FetchInstanceMetadataService.ts
@@ -15,6 +15,7 @@ import { LoggerService } from '@/core/LoggerService.js';
 import { HttpRequestService } from '@/core/HttpRequestService.js';
 import { bindThis } from '@/decorators.js';
 import { FederatedInstanceService } from '@/core/FederatedInstanceService.js';
+import { REMOTE_SERVER_CACHE_TTL } from '@/const.js';
 import type { DOMWindow } from 'jsdom';
 
 type NodeInfo = {
@@ -24,6 +25,7 @@ type NodeInfo = {
 		version?: unknown;
 	};
 	metadata?: {
+		httpMessageSignaturesImplementationLevel?: unknown,
 		name?: unknown;
 		nodeName?: unknown;
 		nodeDescription?: unknown;
@@ -39,6 +41,7 @@ type NodeInfo = {
 @Injectable()
 export class FetchInstanceMetadataService {
 	private logger: Logger;
+	private httpColon = 'https://';
 
 	constructor(
 		private httpRequestService: HttpRequestService,
@@ -48,6 +51,7 @@ export class FetchInstanceMetadataService {
 		private redisClient: Redis.Redis,
 	) {
 		this.logger = this.loggerService.getLogger('metadata', 'cyan');
+		this.httpColon = process.env.MISSKEY_USE_HTTP?.toLowerCase() === 'true' ? 'http://' : 'https://';
 	}
 
 	@bindThis
@@ -59,7 +63,7 @@ export class FetchInstanceMetadataService {
 		return await this.redisClient.set(
 			`fetchInstanceMetadata:mutex:v2:${host}`, '1',
 			'EX', 30, // 30秒したら自動でロック解除 https://github.com/misskey-dev/misskey/issues/13506#issuecomment-1975375395
-			'GET' // 古い値を返す（なかったらnull）
+			'GET', // 古い値を返す（なかったらnull）
 		);
 	}
 
@@ -73,23 +77,24 @@ export class FetchInstanceMetadataService {
 	public async fetchInstanceMetadata(instance: MiInstance, force = false): Promise<void> {
 		const host = instance.host;
 
-		// finallyでunlockされてしまうのでtry内でロックチェックをしない
-		// （returnであってもfinallyは実行される）
-		if (!force && await this.tryLock(host) === '1') {
-			// 1が返ってきていたらロックされているという意味なので、何もしない
-			return;
-		}
+		if (!force) {
+			// キャッシュ有効チェックはロック取得前に行う
+			const _instance = await this.federatedInstanceService.fetch(host);
+			const now = Date.now();
+			if (_instance && _instance.infoUpdatedAt != null && (now - _instance.infoUpdatedAt.getTime() < REMOTE_SERVER_CACHE_TTL)) {
+				this.logger.debug(`Skip because updated recently ${_instance.infoUpdatedAt.toJSON()}`);
+				return;
+			}
 
-		try {
-			if (!force) {
-				const _instance = await this.federatedInstanceService.fetch(host);
-				const now = Date.now();
-				if (_instance && _instance.infoUpdatedAt && (now - _instance.infoUpdatedAt.getTime() < 1000 * 60 * 60 * 24)) {
-					// unlock at the finally caluse
-					return;
-				}
+			// finallyでunlockされてしまうのでtry内でロックチェックをしない
+			// （returnであってもfinallyは実行される）
+			if (await this.tryLock(host) === '1') {
+				// 1が返ってきていたら他にロックされているという意味なので、何もしない
+				return;
 			}
+		}
 
+		try {
 			this.logger.info(`Fetching metadata of ${instance.host} ...`);
 
 			const [info, dom, manifest] = await Promise.all([
@@ -118,6 +123,14 @@ export class FetchInstanceMetadataService {
 				updates.openRegistrations = info.openRegistrations;
 				updates.maintainerName = info.metadata ? info.metadata.maintainer ? (info.metadata.maintainer.name ?? null) : null : null;
 				updates.maintainerEmail = info.metadata ? info.metadata.maintainer ? (info.metadata.maintainer.email ?? null) : null : null;
+				if (info.metadata && info.metadata.httpMessageSignaturesImplementationLevel && (
+					info.metadata.httpMessageSignaturesImplementationLevel === '01' ||
+					info.metadata.httpMessageSignaturesImplementationLevel === '11'
+				)) {
+					updates.httpMessageSignaturesImplementationLevel = info.metadata.httpMessageSignaturesImplementationLevel;
+				} else {
+					updates.httpMessageSignaturesImplementationLevel = '00';
+				}
 			}
 
 			if (name) updates.name = name;
@@ -129,6 +142,12 @@ export class FetchInstanceMetadataService {
 			await this.federatedInstanceService.update(instance.id, updates);
 
 			this.logger.succ(`Successfuly updated metadata of ${instance.host}`);
+			this.logger.debug('Updated metadata:', {
+				info: !!info,
+				dom: !!dom,
+				manifest: !!manifest,
+				updates,
+			});
 		} catch (e) {
 			this.logger.error(`Failed to update metadata of ${instance.host}: ${e}`);
 		} finally {
@@ -141,7 +160,7 @@ export class FetchInstanceMetadataService {
 		this.logger.info(`Fetching nodeinfo of ${instance.host} ...`);
 
 		try {
-			const wellknown = await this.httpRequestService.getJson('https://' + instance.host + '/.well-known/nodeinfo')
+			const wellknown = await this.httpRequestService.getJson(this.httpColon + instance.host + '/.well-known/nodeinfo')
 				.catch(err => {
 					if (err.statusCode === 404) {
 						throw new Error('No nodeinfo provided');
@@ -184,7 +203,7 @@ export class FetchInstanceMetadataService {
 	private async fetchDom(instance: MiInstance): Promise<DOMWindow['document']> {
 		this.logger.info(`Fetching HTML of ${instance.host} ...`);
 
-		const url = 'https://' + instance.host;
+		const url = this.httpColon + instance.host;
 
 		const html = await this.httpRequestService.getHtml(url);
 
@@ -196,7 +215,7 @@ export class FetchInstanceMetadataService {
 
 	@bindThis
 	private async fetchManifest(instance: MiInstance): Promise<Record<string, unknown> | null> {
-		const url = 'https://' + instance.host;
+		const url = this.httpColon + instance.host;
 
 		const manifestUrl = url + '/manifest.json';
 
@@ -207,7 +226,7 @@ export class FetchInstanceMetadataService {
 
 	@bindThis
 	private async fetchFaviconUrl(instance: MiInstance, doc: DOMWindow['document'] | null): Promise<string | null> {
-		const url = 'https://' + instance.host;
+		const url = this.httpColon + instance.host;
 
 		if (doc) {
 			// https://github.com/misskey-dev/misskey/pull/8220#issuecomment-1025104043
@@ -234,12 +253,12 @@ export class FetchInstanceMetadataService {
 	@bindThis
 	private async fetchIconUrl(instance: MiInstance, doc: DOMWindow['document'] | null, manifest: Record<string, any> | null): Promise<string | null> {
 		if (manifest && manifest.icons && manifest.icons.length > 0 && manifest.icons[0].src) {
-			const url = 'https://' + instance.host;
+			const url = this.httpColon + instance.host;
 			return (new URL(manifest.icons[0].src, url)).href;
 		}
 
 		if (doc) {
-			const url = 'https://' + instance.host;
+			const url = this.httpColon + instance.host;
 
 			// https://github.com/misskey-dev/misskey/pull/8220#issuecomment-1025104043
 			const links = Array.from(doc.getElementsByTagName('link')).reverse();