config's strange behavior | NestJS | Page 1

cerulean dirge Sep 21, 2023, 1:58 PM

#

import { KafkaOptions, Transport } from '@nestjs/microservices';
import { KafkaConfig, logLevel } from '@nestjs/microservices/external/kafka.interface';
import { ConfigService } from '@nestjs/config';

export const createConfig = (config: ConfigService): KafkaOptions => {
let clientConfig: KafkaConfig = {
ssl: config.get<boolean>('kafka.ssl'),
brokers: config
.get<string>('kafka.brokers')
.split(',')
.map((b) => b.trim()),
logLevel: config.get<logLevel>('kafka.clientLogLevel'),
};
}

I just set a variable in env like KAFKA_BROKERS and got a value
Does config convert environment variables？

cobalt quiver Sep 21, 2023, 1:59 PM

#

You may need to provide a bit moe context for your question

cerulean dirge Sep 21, 2023, 3:48 PM

#

@cobalt quiver The project needs to set up nest microservices kafka configuration, it is deployed in k8s, set configmap KAFKA_XXXX, and I found that the code to get config is like this, it gets the value that I set, but I think it is impossible to get, Unless nest config converts KAFKA_XXXX to kafka.xxxx

#

Will nest config do this conversion from AA_BB to aa.bb？

cobalt quiver Sep 21, 2023, 3:50 PM

#

I don't follow why it's impossible/what the problem is here. config.get(key) is to get the process.env value for key. If you set the key as KAFKA_XXXX in the .env/process.env then you need to pass that same KAFKA_XXXX to the config.get() method

#

Whatever the value is, that's what Nest will return

cerulean dirge Sep 21, 2023, 3:51 PM

#

The code says kafka.brokers, but I didn't set kafka.brokers

cobalt quiver Sep 21, 2023, 3:51 PM

#

Just like an object, using string access indicies right?

const obj = {
  foo: 'FOO',
  BAR: 'bar'
}
obj['foo'] === 'FOO'
obj['FOO'] === undefined
obj['BAR'] === 'bar'
obj['bar'] === undefined

cobalt quiver Sep 21, 2023, 3:51 PM

#

cerulean dirge The code says kafka.brokers, but I didn't set kafka.brokers

Then it'll return undefined and you need to change it to a proper key

cerulean dirge Sep 21, 2023, 3:53 PM

#

i think is
const myvalue = {
'AA_BB': '123'
}

nestConfig.get('AA_BB') === '123'

#

but code is like
const myvalue = {
'AA_BB': '123'
}
nestConfig.get('aa.bb') === '123'

cobalt quiver Sep 21, 2023, 3:53 PM

#

If that's how you've got the process.env set, then yeah makes sense

cobalt quiver Sep 21, 2023, 3:54 PM

#

cerulean dirge but code is like const myvalue = { 'AA_BB': '123' } nestConfig.get('aa.bb') =...

That doesn't make sense. Nest doesn't change the keys

cerulean dirge Sep 21, 2023, 3:54 PM

#

That's where I get confused

cobalt quiver Sep 21, 2023, 3:55 PM

#

Does k8s configmap duplicate the keys as upper and lowercase?

cerulean dirge Sep 21, 2023, 3:56 PM

#

I don't think I went inside the corresponding pod and ran env cmd and didn't see 'aa.bb

#

Wow, you're the author of nest-commander. That's great！

cobalt quiver Sep 21, 2023, 4:01 PM

#

Hmm, I don't see anything where it would automatically change the case of the key. Can you get us a reproduction that shows this? I'd like to document it if that's the case, but I don't think it should be

cerulean dirge Sep 21, 2023, 4:07 PM

#

import { KafkaOptions, Transport } from '@maiden ermine/microservices';
import { KafkaConfig, logLevel } from '@nestjs/microservices/external/kafka.interface';
import { ConfigService } from '@maiden ermine/config';

export const createConfig = (config: ConfigService): KafkaOptions => {
let clientConfig: KafkaConfig = {
ssl: config.get<boolean>('kafka.ssl'),
brokers: config
.get<string>('kafka.brokers')
.split(',')
.map((b) => b.trim()),
logLevel: config.get<logLevel>('kafka.clientLogLevel'),
};

const password = config.get<string>('kafka.password');
const username = config.get<string>('kafka.username');

if (username && password) {
clientConfig = {
...clientConfig,
clientId: config.get<string>('kafka.clientId'),
sasl: {
mechanism: 'scram-sha-512',
username: username,
password: password,
},
};
}

return {
transport: Transport.KAFKA,
options: {
client: clientConfig,
consumer: {
groupId: config.get<string>('kafka.consumerGroupId'),
},
},
};
};

#

kafka-transport-config.ts

#

import { Module, Provider } from '@maiden ermine/common';
import { ConfigService } from '@maiden ermine/config';
import { createConfig } from './kafka-transport-config';
import { ClientProxyFactory } from '@maiden ermine/microservices';

export const KAFKA_CLIENT = 'KAFKA_CLIENT';
const proxyFactory: Provider = {
provide: KAFKA_CLIENT,
useFactory: (configService: ConfigService) => {
const configs = createConfig(configService);
return ClientProxyFactory.create(configs);
},
inject: [ConfigService],
};
@Module({
providers: [proxyFactory],
exports: [proxyFactory],
})
export class KafkaTransportConfigModule {}

#

kafka-transport-config.module.ts

#

export * from './kafka-transport-config.module';
export * from './kafka-transport-config';

#

index.ts

cobalt quiver Sep 21, 2023, 4:08 PM

#

A git repo would be better than the giant code blocks

cerulean dirge Sep 21, 2023, 4:09 PM

#

The code for this little lib is like this

#

sorry I can't put it in git

cobalt quiver Sep 21, 2023, 4:10 PM

#

cerulean dirge sorry I can't put it in git

Then how am I supposed to reproduce it?

cerulean dirge Sep 21, 2023, 4:10 PM

#

Except for the config.get<boolean>('kafka.ssl') part, it's pretty generic

#

Well, I'll create a project, which may take some time, or I'll let you know when I find out why

#

Thank you so much for making me feel like I'm not fighting alone

cerulean dirge Sep 21, 2023, 4:33 PM

#

@cobalt quiver

#

import { Module } from '@maiden ermine/common';
import { ConfigModule } from '@maiden ermine/config';

export const configuration = () => ({
kafka: {
brokers: process.env.KAFKA_BROKERS || 'localhost:29092',
clientId: process.env.KAFKA_CLIENT_ID || 'xxx-analytics',
username: process.env.KAFKA_USERNAME,
password: process.env.KAFKA_PASSWORD,
ssl: process.env.KAFKA_SSL === 'true',
consumerGroupId: process.env.KAFKA_CONSUMER_GROUP_ID || 'xxx-xxxx',
clientLogLevel: logLevel[process.env.KAFKA_CONSUMER_LOG_LEVEL] || logLevel.INFO,
}
});

@Module({
imports: [
ConfigModule.forRoot({
isGlobal: true,
load: [configuration],
}),
],
providers: [],
exports: [],
})
export class myConfigModule {}

// Usage
import { myConfigModule } from '@app/config';
@Module({
imports: [
ConfigModule.forRoot(),
myConfigModule,
...
})

#

Hi, I found the reason, the project set this up in front!

cobalt quiver Sep 21, 2023, 4:34 PM

#

Ah, you're using configuration namespacing. So yeah, you can use kakfa.brokers which Nest now knows points to process.env.KAFKA_BROKERS which is why it works the way you're seeing

#config's strange behavior