perf(federation): Ed25519署名に対応する (#13464)

* 1. ed25519キーペアを発行・Personとして公開鍵を送受信

* validate additionalPublicKeys

* getAuthUserFromApIdはmainを選ぶ

* :v:

* fix

* signatureAlgorithm

* set publicKeyCache lifetime

* refresh

* httpMessageSignatureAcceptable

* ED25519_SIGNED_ALGORITHM

* ED25519_PUBLIC_KEY_SIGNATURE_ALGORITHM

* remove sign additionalPublicKeys signature requirements

* httpMessageSignaturesSupported

* httpMessageSignaturesImplementationLevel

* httpMessageSignaturesImplementationLevel: '01'

* perf(federation): Use hint for getAuthUserFromApId (#13470)

* Hint for getAuthUserFromApId

* とどのつまりこれでいいのか？

* use @misskey-dev/node-http-message-signatures

* fix

* signedPost, signedGet

* ap-request.tsを復活させる

* remove digest prerender

* fix test?

* fix test

* add httpMessageSignaturesImplementationLevel to FederationInstance

* ManyToOne

* fetchPersonWithRenewal

* exactKey

* :v:

* use const

* use gen-key-pair fn. from  '@misskey-dev/node-http-message-signatures'

* update node-http-message-signatures

* fix

* @misskey-dev/node-http-message-signatures@0.0.0-alpha.11

* getAuthUserFromApIdでupdatePersonの頻度を増やす

* cacheRaw.date

* use requiredInputs
https://github.com/misskey-dev/misskey/pull/13464#discussion_r1509964359

* update @misskey-dev/node-http-message-signatures

* clean up

* err msg

* fix(backend): fetchInstanceMetadataのLockが永遠に解除されない問題を修正

Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com>

* fix httpMessageSignaturesImplementationLevel validation

* fix test

* fix

* comment

* comment

* improve test

* fix

* use Promise.all in genRSAAndEd25519KeyPair

* refreshAndprepareEd25519KeyPair

* refreshAndfindKey

* commetn

* refactor public keys add

* digestプリレンダを復活させる

RFC実装時にどうするか考える

* fix, async

* fix

* !== true

* use save

* Deliver update person when new key generated (not tested)
https://github.com/misskey-dev/misskey/pull/13464#issuecomment-1977049061

* 循環参照で落ちるのを解消？

* fix?

* Revert "fix?"

This reverts commit 0082f6f8e8c5d5febd14933ba9a1ac643f70ca92.

* a

* logger

* log

* change logger

* 秘密鍵の変更は、フラグではなく鍵を引き回すようにする

* addAllKnowingSharedInboxRecipe

* nanka meccha kaeta

* delivre

* キャッシュ有効チェックはロック取得前に行う

* @misskey-dev/node-http-message-signatures@0.0.3

* PrivateKeyPem

* getLocalUserPrivateKey

* fix test

* if

* fix ap-request

* update node-http-message-signatures

* fix type error

* update package

* fix type

* update package

* retry no key

* @misskey-dev/node-http-message-signatures@0.0.8

* fix type error

* log keyid

* logger

* db-resolver

* JSON.stringify

* HTTP Signatureがなかったり使えなかったりしそうな場合にLD Signatureを活用するように

* inbox-delayed use actor if no signature

* ユーザーとキーの同一性チェックはhostの一致にする

* log signature parse err

* save array

* とりあえずtryで囲っておく

* fetchPersonWithRenewalでエラーが起きたら古いデータを返す

* use transactionalEntityManager

* fix spdx

* @misskey-dev/node-http-message-signatures@0.0.10

* add comment

* fix

* publicKeyに配列が入ってもいいようにする
https://github.com/misskey-dev/misskey/pull/13950

* define additionalPublicKeys

* fix

* merge fix

* refreshAndprepareEd25519KeyPair → refreshAndPrepareEd25519KeyPair

* remove gen-key-pair.ts

* defaultMaxListeners = 512

* Revert "defaultMaxListeners = 512"

This reverts commit f2c412c18057a9300540794ccbe4dfbf6d259ed6.

* genRSAAndEd25519KeyPairではキーを直列に生成する?

* maxConcurrency: 8

* maxConcurrency: 16

* maxConcurrency: 8

* Revert "genRSAAndEd25519KeyPairではキーを直列に生成する?"

This reverts commit d0aada55c1ed5aa98f18731ec82f3ac5eb5a6c16.

* maxWorkers: '90%'

* Revert "maxWorkers: '90%'"

This reverts commit 9e0a93f110456320d6485a871f014f7cdab29b33.

* e2e/timelines.tsで個々のテストに対するtimeoutを削除, maxConcurrency: 32

* better error handling of this.userPublickeysRepository.delete

* better comment

* set result to keypairEntityCache

* deliverJobConcurrency: 16, deliverJobPerSec: 1024, inboxJobConcurrency: 4

* inboxJobPerSec: 64

* delete request.headers['host'];

* fix

* // node-fetch will generate this for us. if we keep 'Host', it won't change with redirects!

* move delete host

* modify comment

* modify comment

* fix correct → collect

* refreshAndfindKey → refreshAndFindKey

* modify comment

* modify attachLdSignature

* getApId, InboxProcessorService

* TODO

* [skip ci] add CHANGELOG

---------

Co-authored-by: MeiMei <30769358+mei23@users.noreply.github.com>
Co-authored-by: まっちゃとーにゅ <17376330+u1-liquid@users.noreply.github.com>

2 files changed, 97 insertions, 71 deletions

diff --git a/packages/backend/src/queue/processors/DeliverProcessorService.ts b/packages/backend/src/queue/processors/DeliverProcessorService.ts
index d665945861..3bd9187e8b 100644
--- a/packages/backend/src/queue/processors/DeliverProcessorService.ts
+++ b/packages/backend/src/queue/processors/DeliverProcessorService.ts
@@ -73,25 +73,33 @@ export class DeliverProcessorService {
 		}
 
 		try {
-			await this.apRequestService.signedPost(job.data.user, job.data.to, job.data.content, job.data.digest);
+			const _server = await this.federatedInstanceService.fetch(host);
+			await this.fetchInstanceMetadataService.fetchInstanceMetadata(_server).then(() => {});
+			const server = await this.federatedInstanceService.fetch(host);
+
+			await this.apRequestService.signedPost(
+				job.data.user,
+				job.data.to,
+				job.data.content,
+				server.httpMessageSignaturesImplementationLevel,
+				job.data.digest,
+				job.data.privateKey,
+			);
 
 			// Update stats
-			this.federatedInstanceService.fetch(host).then(i => {
-				if (i.isNotResponding) {
-					this.federatedInstanceService.update(i.id, {
-						isNotResponding: false,
-						notRespondingSince: null,
-					});
-				}
+			if (server.isNotResponding) {
+				this.federatedInstanceService.update(server.id, {
+					isNotResponding: false,
+					notRespondingSince: null,
+				});
+			}
 
-				this.fetchInstanceMetadataService.fetchInstanceMetadata(i);
-				this.apRequestChart.deliverSucc();
-				this.federationChart.deliverd(i.host, true);
+			this.apRequestChart.deliverSucc();
+			this.federationChart.deliverd(server.host, true);
 
-				if (meta.enableChartsForFederatedInstances) {
-					this.instanceChart.requestSent(i.host, true);
-				}
-			});
+			if (meta.enableChartsForFederatedInstances) {
+				this.instanceChart.requestSent(server.host, true);
+			}
 
 			return 'Success';
 		} catch (res) {
diff --git a/packages/backend/src/queue/processors/InboxProcessorService.ts b/packages/backend/src/queue/processors/InboxProcessorService.ts
index fa7009f8f5..935c623df1 100644
--- a/packages/backend/src/queue/processors/InboxProcessorService.ts
+++ b/packages/backend/src/queue/processors/InboxProcessorService.ts
@@ -5,8 +5,8 @@
 
 import { URL } from 'node:url';
 import { Injectable } from '@nestjs/common';
-import httpSignature from '@peertube/http-signature';
 import * as Bull from 'bullmq';
+import { verifyDraftSignature } from '@misskey-dev/node-http-message-signatures';
 import type Logger from '@/logger.js';
 import { MetaService } from '@/core/MetaService.js';
 import { FederatedInstanceService } from '@/core/FederatedInstanceService.js';
@@ -20,6 +20,7 @@ import type { MiRemoteUser } from '@/models/User.js';
 import type { MiUserPublickey } from '@/models/UserPublickey.js';
 import { ApDbResolverService } from '@/core/activitypub/ApDbResolverService.js';
 import { StatusError } from '@/misc/status-error.js';
+import * as Acct from '@/misc/acct.js';
 import { UtilityService } from '@/core/UtilityService.js';
 import { ApPersonService } from '@/core/activitypub/models/ApPersonService.js';
 import { JsonLdService } from '@/core/activitypub/JsonLdService.js';
@@ -52,8 +53,15 @@ export class InboxProcessorService {
 
 	@bindThis
 	public async process(job: Bull.Job<InboxJobData>): Promise<string> {
-		const signature = job.data.signature;	// HTTP-signature
+		const signature = job.data.signature ?
+			'version' in job.data.signature ? job.data.signature.value : job.data.signature
+			: null;
+		if (Array.isArray(signature)) {
+			// RFC 9401はsignatureが配列になるが、とりあえずエラーにする
+			throw new Error('signature is array');
+		}
 		let activity = job.data.activity;
+		let actorUri = getApId(activity.actor);
 
 		//#region Log
 		const info = Object.assign({}, activity);
@@ -61,7 +69,7 @@ export class InboxProcessorService {
 		this.logger.debug(JSON.stringify(info, null, 2));
 		//#endregion
 
-		const host = this.utilityService.toPuny(new URL(signature.keyId).hostname);
+		const host = this.utilityService.toPuny(new URL(actorUri).hostname);
 
 		// ブロックしてたら中断
 		const meta = await this.metaService.fetch();
@@ -69,69 +77,76 @@ export class InboxProcessorService {
 			return `Blocked request: ${host}`;
 		}
 
-		const keyIdLower = signature.keyId.toLowerCase();
-		if (keyIdLower.startsWith('acct:')) {
-			return `Old keyId is no longer supported. ${keyIdLower}`;
-		}
-
 		// HTTP-Signature keyIdを元にDBから取得
-		let authUser: {
-			user: MiRemoteUser;
-			key: MiUserPublickey | null;
-		} | null = await this.apDbResolverService.getAuthUserFromKeyId(signature.keyId);
+		let authUser: Awaited<ReturnType<typeof this.apDbResolverService.getAuthUserFromApId>> = null;
+		let httpSignatureIsValid = null as boolean | null;
 
-		// keyIdでわからなければ、activity.actorを元にDBから取得 || activity.actorを元にリモートから取得
-		if (authUser == null) {
-			try {
-				authUser = await this.apDbResolverService.getAuthUserFromApId(getApId(activity.actor));
-			} catch (err) {
-				// 対象が4xxならスキップ
-				if (err instanceof StatusError) {
-					if (!err.isRetryable) {
-						throw new Bull.UnrecoverableError(`skip: Ignored deleted actors on both ends ${activity.actor} - ${err.statusCode}`);
-					}
-					throw new Error(`Error in actor ${activity.actor} - ${err.statusCode}`);
+		try {
+			authUser = await this.apDbResolverService.getAuthUserFromApId(actorUri, signature?.keyId);
+		} catch (err) {
+			// 対象が4xxならスキップ
+			if (err instanceof StatusError) {
+				if (!err.isRetryable) {
+					throw new Bull.UnrecoverableError(`skip: Ignored deleted actors on both ends ${activity.actor} - ${err.statusCode}`);
 				}
+				throw new Error(`Error in actor ${activity.actor} - ${err.statusCode}`);
 			}
 		}
 
-		// それでもわからなければ終了
-		if (authUser == null) {
+		// authUser.userがnullならスキップ
+		if (authUser != null && authUser.user == null) {
 			throw new Bull.UnrecoverableError('skip: failed to resolve user');
 		}
 
-		// publicKey がなくても終了
-		if (authUser.key == null) {
-			throw new Bull.UnrecoverableError('skip: failed to resolve user publicKey');
+		if (signature != null && authUser != null) {
+			if (signature.keyId.toLowerCase().startsWith('acct:')) {
+				this.logger.warn(`Old keyId is no longer supported. lowerKeyId=${signature.keyId.toLowerCase()}`);
+			} else if (authUser.key != null) {
+				// keyがなかったらLD Signatureで検証するべき
+				// HTTP-Signatureの検証
+				const errorLogger = (ms: any) => this.logger.error(ms);
+				httpSignatureIsValid = await verifyDraftSignature(signature, authUser.key.keyPem, errorLogger);
+				this.logger.debug('Inbox message validation: ', {
+					userId: authUser.user.id,
+					userAcct: Acct.toString(authUser.user),
+					parsedKeyId: signature.keyId,
+					foundKeyId: authUser.key.keyId,
+					httpSignatureValid: httpSignatureIsValid,
+				});
+			}
 		}
 
-		// HTTP-Signatureの検証
-		const httpSignatureValidated = httpSignature.verifySignature(signature, authUser.key.keyPem);
-
-		// また、signatureのsignerは、activity.actorと一致する必要がある
-		if (!httpSignatureValidated || authUser.user.uri !== activity.actor) {
+		if (
+			authUser == null ||
+			httpSignatureIsValid !== true ||
+			authUser.user.uri !== actorUri // 一応チェック
+		) {
 			// 一致しなくても、でもLD-Signatureがありそうならそっちも見る
 			const ldSignature = activity.signature;
-			if (ldSignature) {
+
+			if (ldSignature && ldSignature.creator) {
 				if (ldSignature.type !== 'RsaSignature2017') {
 					throw new Bull.UnrecoverableError(`skip: unsupported LD-signature type ${ldSignature.type}`);
 				}
 
-				// ldSignature.creator: https://example.oom/users/user#main-key
-				// みたいになっててUserを引っ張れば公開キーも入ることを期待する
-				if (ldSignature.creator) {
-					const candicate = ldSignature.creator.replace(/#.*/, '');
-					await this.apPersonService.resolvePerson(candicate).catch(() => null);
+				if (ldSignature.creator.toLowerCase().startsWith('acct:')) {
+					throw new Bull.UnrecoverableError(`old key not supported ${ldSignature.creator}`);
 				}
 
-				// keyIdからLD-Signatureのユーザーを取得
-				authUser = await this.apDbResolverService.getAuthUserFromKeyId(ldSignature.creator);
+				authUser = await this.apDbResolverService.getAuthUserFromApId(actorUri, ldSignature.creator);
+
 				if (authUser == null) {
-					throw new Bull.UnrecoverableError('skip: LD-Signatureのユーザーが取得できませんでした');
+					throw new Bull.UnrecoverableError(`skip: LD-Signatureのactorとcreatorが一致しませんでした uri=${actorUri} creator=${ldSignature.creator}`);
+				}
+				if (authUser.user == null) {
+					throw new Bull.UnrecoverableError(`skip: LD-Signatureのユーザーが取得できませんでした uri=${actorUri} creator=${ldSignature.creator}`);
+				}
+				// 一応actorチェック
+				if (authUser.user.uri !== actorUri) {
+					throw new Bull.UnrecoverableError(`skip: LD-Signature user(${authUser.user.uri}) !== activity.actor(${actorUri})`);
 				}
-
 				if (authUser.key == null) {
-					throw new Bull.UnrecoverableError('skip: LD-SignatureのユーザーはpublicKeyを持っていませんでした');
+					throw new Bull.UnrecoverableError(`skip: LD-SignatureのユーザーはpublicKeyを持っていませんでした uri=${actorUri} creator=${ldSignature.creator}`);
 				}
 
 				const jsonLd = this.jsonLdService.use();
@@ -142,13 +157,27 @@ export class InboxProcessorService {
 					throw new Bull.UnrecoverableError('skip: LD-Signatureの検証に失敗しました');
 				}
 
+				// ブロックしてたら中断
+				const ldHost = this.utilityService.extractDbHost(authUser.user.uri);
+				if (this.utilityService.isBlockedHost(meta.blockedHosts, ldHost)) {
+					throw new Bull.UnrecoverableError(`Blocked request: ${ldHost}`);
+				}
+
 				// アクティビティを正規化
+				// GHSA-2vxv-pv3m-3wvj
 				delete activity.signature;
 				try {
 					activity = await jsonLd.compact(activity) as IActivity;
 				} catch (e) {
 					throw new Bull.UnrecoverableError(`skip: failed to compact activity: ${e}`);
 				}
+
+				// actorが正規化前後で一致しているか確認
+				actorUri = getApId(activity.actor);
+				if (authUser.user.uri !== actorUri) {
+					throw new Bull.UnrecoverableError(`skip: LD-Signature user(${authUser.user.uri}) !== activity(after normalization).actor(${actorUri})`);
+				}
+
 				// TODO: 元のアクティビティと非互換な形に正規化される場合は転送をスキップする
 				// https://github.com/mastodon/mastodon/blob/664b0ca/app/services/activitypub/process_collection_service.rb#L24-L29
 				activity.signature = ldSignature;
@@ -158,19 +187,8 @@ export class InboxProcessorService {
 				delete compactedInfo['@context'];
 				this.logger.debug(`compacted: ${JSON.stringify(compactedInfo, null, 2)}`);
 				//#endregion
-
-				// もう一度actorチェック
-				if (authUser.user.uri !== activity.actor) {
-					throw new Bull.UnrecoverableError(`skip: LD-Signature user(${authUser.user.uri}) !== activity.actor(${activity.actor})`);
-				}
-
-				// ブロックしてたら中断
-				const ldHost = this.utilityService.extractDbHost(authUser.user.uri);
-				if (this.utilityService.isBlockedHost(meta.blockedHosts, ldHost)) {
-					throw new Bull.UnrecoverableError(`Blocked request: ${ldHost}`);
-				}
 			} else {
-				throw new Bull.UnrecoverableError(`skip: http-signature verification failed and no LD-Signature. keyId=${signature.keyId}`);
+				throw new Bull.UnrecoverableError(`skip: http-signature verification failed and no LD-Signature. http_signature_keyId=${signature?.keyId}`);
 			}
 		}