chore: make vault action consumable (#43)

* chore: make vault action consumable

* fix prefixless queries to default to data

* fix the right build entrypoint

* make output more forgiving and shore up selectors

* clarify doc language

* add npmtoken

.github/workflows/build.yml

@@ -168,4 +168,5 @@ jobs:
       run: npx semantic-release
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
 

.vscode/launch.json

@@ -0,0 +1,35 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "type": "node",
+            "name": "vscode-jest-tests",
+            "request": "launch",
+            "args": [
+                "--runInBand"
+            ],
+            "cwd": "${workspaceFolder}",
+            "console": "integratedTerminal",
+            "internalConsoleOptions": "neverOpen",
+            "disableOptimisticBPs": true,
+            "program": "${workspaceFolder}/node_modules/jest/bin/jest"
+        },        
+        {
+            "type": "node",
+            "name": "vscode-integration-tests",
+            "request": "launch",
+            "args": [
+                "--runInBand",
+                "--config=${workspaceFolder}/integrationTests/basic/jest.config.js"
+            ],
+            "cwd": "${workspaceFolder}",
+            "console": "integratedTerminal",
+            "internalConsoleOptions": "neverOpen",
+            "disableOptimisticBPs": true,
+            "program": "${workspaceFolder}/node_modules/jest/bin/jest"
+        }
+    ]
+}

README.md

@@ -12,6 +12,7 @@ By default, this action pulls from  [Version 2](https://www.vaultproject.io/docs
     - [Simple Key](#simple-key)
     - [Set Output Variable Name](#set-output-variable-name)
     - [Multiple Secrets](#multiple-secrets)
+    - [Nested Secrets](#nested-secrets)
     - [Using K/V version 1](#using-kv-version-1)
 - [Custom K/V Engine Path](#custom-kv-engine-path)
 - [Other Secret Engines](#other-secret-engines)
@@ -20,6 +21,7 @@ By default, this action pulls from  [Version 2](https://www.vaultproject.io/docs
     - [Namespace](#namespace)
 - [Reference](#reference)
 - [Masking - Hiding Secrets from Logs](#masking---hiding-secrets-from-logs)
+- [Normalization](#normalization)
 
 <!-- /TOC -->
 
@@ -81,7 +83,7 @@ The `secrets` parameter is a set of multiple secret requests separated by the `;
 Each secret request is comprised of the `path` and the `key` of the desired secret, and optionally the desired Env Var output name.
 
 ```raw
-{{ Secret Path }} {{ Secret Key }} | {{ Output Variable Name }}
+{{ Secret Path }} {{ Secret Key or Selector }} | {{ Env/Output Variable Name }}
 ```
 
 ### Simple Key
@@ -151,6 +153,21 @@ with:
         ci/aws secretKey | AWS_SECRET_ACCESS_KEY
 ```
 
+### Nested Secrets
+
+By default, `vault-action` will read the key from `data.data` in the response for the K/V v2 engine (default), or `data` for K/V v1 and other Secret Engines (see below for more info).
+If you need to retrieve a sub-key from a JSON document, or are interested in some other component of the Vault response, you can specify a different key as the path to the desired out.
+
+_**Important**_: You must specify an [Output Variable Name](#set-output-variable-name) when using this method.
+
+```yaml
+with:
+    secrets: |
+        my/path pair.key | NESTED_SECRET ;
+```
+
+Under the covers, we're using [JSONata](https://jsonata.org/) to provide the querying functionality. Any valid JSONata syntax is valid here, and will be outputed as a `JSON.stringify`-ied result.
+
 ### Using K/V version 1
 
 By default, `vault-action` expects a K/V engine using [version 2](https://www.vaultproject.io/docs/secrets/kv/kv-v2.html).
@@ -179,7 +196,7 @@ This way, the `ci` secret in the example above will be retrieved from `my-secret
 
 While this action primarily supports the K/V engine, it is possible to request secrets from other engines in Vault.
 
-To do so when specifying the `Secret Path`, just append a leading formard slash (`/`) and specify the path as described in the Vault API documentation.
+To do so when specifying the `Secret Path`, just append a leading forward slash (`/`) and specify the path as described in the Vault API documentation.
 
 For example, to retrieve a stored secret from the [`cubbyhole` engine](https://www.vaultproject.io/api-docs/secret/cubbyhole/), assuming you have a stored secret at the path `foo` with the contents:
 
@@ -231,7 +248,7 @@ with:
 
 would work fine.
 
-NOTE: The `Secret Key` is pulled from the `data` property of the response.
+*NOTE: Per [Nested Secrets](#nested-secrets), the `Key` is pulled from the `data` property of the response.*
 
 ## Adding Extra Headers
 
@@ -247,13 +264,13 @@ with:
       X-Secure-Secret: ${{ secrets.SECURE_SECRET }}
 ```
 
-This will automatically add the `x-secure-id` and `x-secure-secret` headers to every request to vault.
+This will automatically add the `x-secure-id` and `x-secure-secret` headers to every request to Vault.
 
 ## Vault Enterprise Features
 
 ### Namespace
 
-If you need to retrieve secrets from a specific vault namespace, all that's required is an additional parameter specifying the namespace.
+If you need to retrieve secrets from a specific Vault namespace, all that's required is an additional parameter specifying the namespace.
 
 ```yaml
 steps:
@@ -273,7 +290,7 @@ steps:
 
 ## Reference
 
-Here is all the inputs available through `with`:
+Here are all the inputs available through `with`:
 
 | Input          | Description                                                                                                                                          | Default | Required |
 | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
@@ -295,3 +312,7 @@ Here is all the inputs available through `with`:
 
 This action uses GitHub Action's built-in masking, so all variables will automatically be masked (aka hidden) if printed to the console or to logs.
 **This only obscures secrets from output logs.** If someone has the ability to edit your workflows, then they are able to read and therefore write secrets to somewhere else just like normal GitHub Secrets.
+
+## Normalization
+
+To make it simpler to consume certain secrets as env vars, if no Env/Output Var Name is specified `vault-action` will replace and `.` chars with `__`, remove any other non-letter or number characters. If you're concerned about the result, it's recommended to provider an explicit Output Var Key.

package-lock.json

@@ -7693,6 +7693,11 @@
         "minimist": "^1.2.5"
       }
     },
+    "jsonata": {
+      "version": "1.8.2",
+      "resolved": "https://registry.npmjs.org/jsonata/-/jsonata-1.8.2.tgz",
+      "integrity": "sha512-ma5F/Bs47dZfJfDZ0Dt37eIbzVBVKZIDqsZSqdCCAPNHxKn+s3+CfMA6ahVVlf8Y1hyIjXkVLFU7yv4XxRfihA=="
+    },
     "jsonfile": {
       "version": "6.0.1",
       "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.0.1.tgz",

package.json

@@ -4,18 +4,23 @@
     "description": "A Github Action that allows you to consume vault secrets as secure environment variables.",
     "main": "dist/index.js",
     "scripts": {
-        "build": "ncc build src/index.js -o dist",
+        "build": "ncc build src/entry.js -o dist",
         "test": "jest",
         "test:integration:basic": "jest -c integrationTests/basic/jest.config.js",
         "test:integration:enterprise": "jest -c integrationTests/enterprise/jest.config.js",
         "test:e2e": "jest -c integrationTests/e2e/jest.config.js"
     },
+    "files": [
+        "src/**/*",
+        "dist/**/*"
+    ],
     "release": {
         "branch": "master",
         "plugins": [
             "@semantic-release/commit-analyzer",
             "@semantic-release/release-notes-generator",
-            "@semantic-release/github"
+            "@semantic-release/github",
+            "@semantic-release/npm"
         ],
         "ci": false
     },
@@ -38,8 +43,9 @@
     },
     "homepage": "https://github.com/RichiCoder1/vault-action#readme",
     "dependencies": {
+        "@actions/core": "^1.2.3",
         "got": "^10.2.2",
-        "@actions/core": "^1.2.3"
+        "jsonata": "^1.8.2"
     },
     "devDependencies": {
         "@types/got": "^9.6.9",

src/action.js

@@ -2,7 +2,8 @@
 const core = require('@actions/core');
 const command = require('@actions/core/lib/command');
 const got = require('got').default;
-const { retrieveToken } = require('./auth');
+const jsonata = require('jsonata');
+const { auth: { retrieveToken }, secrets: { getSecrets } } = require('./index');
 
 const AUTH_METHODS = ['approle', 'token', 'github'];
 const VALID_KV_VERSION = [-1, 1, 2];
@@ -39,8 +40,9 @@ async function exportSecrets() {
         defaultOptions.headers["X-Vault-Namespace"] = vaultNamespace;
     }
 
+    const vaultToken = await retrieveToken(vaultMethod, got.extend(defaultOptions));
+    defaultOptions.headers['X-Vault-Token'] = vaultToken;
     const client = got.extend(defaultOptions);
-    const vaultToken = await retrieveToken(vaultMethod, client);
 
     if (!enginePath) {
         enginePath = 'secret';
@@ -55,62 +57,42 @@ async function exportSecrets() {
         throw Error(`You must provide a valid K/V version (${VALID_KV_VERSION.slice(1).join(', ')}). Input: "${kvVersion}"`);
     }
 
-    const responseCache = new Map();
+    const requests = secretRequests.map(request => {
-    for (const secretRequest of secretRequests) {
+        const { path, selector } = request;
-        const { secretPath, outputVarName, envVarName, secretSelector, isJSONPath } = secretRequest;
-        const requestOptions = {
-            headers: {
-                'X-Vault-Token': vaultToken
-            },
-        };
 
-        for (const [headerName, headerValue] of extraHeaders) {
+        if (path.startsWith('/')) {
-            requestOptions.headers[headerName] = headerValue;
+            return request;
         }
+        const kvPath = (kvVersion === 2)
+            ? `/${enginePath}/data/${path}`
+            : `/${enginePath}/${path}`;
+        const kvSelector = (kvVersion === 2)
+            ? `data.data.${selector}`
+            : `data.${selector}`;
+        return { ...request, path: kvPath, selector: kvSelector };
+    });
 
-        if (vaultNamespace != null) {
+    const results = await getSecrets(requests, client);
-            requestOptions.headers["X-Vault-Namespace"] = vaultNamespace;
-        }
 
-        let requestPath = `v1`;
+    for (const result of results) {
-        const kvRequest = !secretPath.startsWith('/')
+        const { value, request, cachedResponse } = result;
-        if (!kvRequest) {
+        if (cachedResponse) {
-            requestPath += secretPath;
-        } else {
-           requestPath += (kvVersion === 2)
-                    ? `/${enginePath}/data/${secretPath}`
-                    : `/${enginePath}/${secretPath}`;
-        }
-
-        let body;
-        if (responseCache.has(requestPath)) {
-            body = responseCache.get(requestPath);
             core.debug('ℹ using cached response');
-        } else {
+        }        
-            const result = await client.get(requestPath, requestOptions);
-            body = result.body;
-            responseCache.set(requestPath, body);
-        }
-
-        let dataDepth = isJSONPath === true ? 0 : kvRequest === false ? 1 : kvVersion;
-
-        const secretData = getResponseData(body, dataDepth);
-        const value = selectData(secretData, secretSelector, isJSONPath);
         command.issue('add-mask', value);
         if (exportEnv) {
-            core.exportVariable(envVarName, `${value}`);
+            core.exportVariable(request.envVarName, `${value}`);
         }
-        core.setOutput(outputVarName, `${value}`);
+        core.setOutput(request.outputVarName, `${value}`);
-        core.debug(`✔ ${secretPath} => outputs.${outputVarName}${exportEnv ? ` | env.${envVarName}` : ''}`);
+        core.debug(`✔ ${request.path} => outputs.${request.outputVarName}${exportEnv ? ` | env.${request.envVarName}` : ''}`);
     }
 };
 
 /** @typedef {Object} SecretRequest 
- * @property {string} secretPath
+ * @property {string} path
  * @property {string} envVarName
  * @property {string} outputVarName
- * @property {string} secretSelector
+ * @property {string} selector
- * @property {boolean} isJSONPath
 */
 
 /**
@@ -127,12 +109,12 @@ function parseSecretsInput(secretsInput) {
     /** @type {SecretRequest[]} */
     const output = [];
     for (const secret of secrets) {
-        let path = secret;
+        let pathSpec = secret;
         let outputVarName = null;
 
         const renameSigilIndex = secret.lastIndexOf('|');
         if (renameSigilIndex > -1) {
-            path = secret.substring(0, renameSigilIndex).trim();
+            pathSpec = secret.substring(0, renameSigilIndex).trim();
             outputVarName = secret.substring(renameSigilIndex + 1).trim();
 
             if (outputVarName.length < 1) {
@@ -140,7 +122,7 @@ function parseSecretsInput(secretsInput) {
             }
         }
 
-        const pathParts = path
+        const pathParts = pathSpec
             .split(/\s+/)
             .map(part => part.trim())
             .filter(part => part.length !== 0);
@@ -149,64 +131,43 @@ function parseSecretsInput(secretsInput) {
             throw Error(`You must provide a valid path and key. Input: "${secret}"`);
         }
 
-        const [secretPath, secretSelector] = pathParts;
+        const [path, selector] = pathParts;
 
-        const isJSONPath = secretSelector.includes('.');
+        /** @type {any} */
+        const selectorAst = jsonata(selector).ast();
 
-        if (isJSONPath && !outputVarName) {
+        if ((selectorAst.type !== "path" || selectorAst.steps[0].stages) && !outputVarName) {
             throw Error(`You must provide a name for the output key when using json selectors. Input: "${secret}"`);
         }
 
         let envVarName = outputVarName;
         if (!outputVarName) {
-            outputVarName = secretSelector;
+            outputVarName = normalizeOutputKey(selector);
-            envVarName = normalizeOutputKey(outputVarName);
+            envVarName = normalizeOutputKey(selector, true);
         }
 
         output.push({
-            secretPath,
+            path,
             envVarName,
             outputVarName,
-            secretSelector,
+            selector
-            isJSONPath
         });
     }
     return output;
 }
 
 /**
- * Parses a JSON response and returns the secret data
+ * Replaces any dot chars to __ and removes non-ascii charts
- * @param {string} responseBody
- * @param {number} dataLevel
- */
-function getResponseData(responseBody, dataLevel) {
-    let secretData = JSON.parse(responseBody);
-
-    for (let i = 0; i < dataLevel; i++) {
-        secretData = secretData['data'];
-    }
-    return secretData;
-}
-
-/**
- * Parses a JSON response and returns the secret data
- * @param {Object} data
- * @param {string} selector
- */
-function selectData(data, selector, isJSONPath) {
-    if (!isJSONPath) {
-        return data[selector];
-    }
-
-    // TODO: JSONPath
-}
-
-/**
- * Replaces any forward-slash characters to
  * @param {string} dataKey
+ * @param {boolean=} isEnvVar
  */
-function normalizeOutputKey(dataKey) {
+function normalizeOutputKey(dataKey, isEnvVar = false) {
-    return dataKey.replace('/', '__').replace(/[^\w-]/, '').toUpperCase();
+    let outputKey = dataKey
+        .replace('.', '__').replace(/[^\p{L}\p{N}_-]/gu, '');
+    if (isEnvVar) {
+        outputKey = outputKey.toUpperCase();
+    }
+    return outputKey;
 }
 
 /**
@@ -237,7 +198,6 @@ function parseHeadersInput(inputKey, inputOptions) {
 module.exports = {
     exportSecrets,
     parseSecretsInput,
-    parseResponse: getResponseData,
     normalizeOutputKey,
     parseHeadersInput
-};
+};

src/action.test.js

@@ -17,11 +17,10 @@ describe('parseSecretsInput', () => {
     it('parses simple secret', () => {
         const output = parseSecretsInput('test key');
         expect(output).toContainEqual({
-            secretPath: 'test',
+            path: 'test',
-            secretSelector: 'key',
+            selector: 'key',
             outputVarName: 'key',
-            envVarName: 'KEY',
+            envVarName: 'KEY'
-            isJSONPath: false
         });
     });
 
@@ -49,10 +48,10 @@ describe('parseSecretsInput', () => {
 
         expect(output).toHaveLength(2);
         expect(output[0]).toMatchObject({
-            secretPath: 'first',
+            path: 'first',
         });
         expect(output[1]).toMatchObject({
-            secretPath: 'second',
+            path: 'second',
         });
     });
 
@@ -78,7 +77,7 @@ describe('parseSecretsInput', () => {
 
         expect(output).toHaveLength(3);
         expect(output[0]).toMatchObject({
-            secretPath: 'first',
+            path: 'first',
         });
         expect(output[1]).toMatchObject({
             outputVarName: 'b',
@@ -88,7 +87,7 @@ describe('parseSecretsInput', () => {
             outputVarName: 'SOME_C',
             envVarName: 'SOME_C',
         });
-    })
+    });
 });
 
 describe('parseHeaders', () => {
@@ -129,41 +128,8 @@ describe('parseHeaders', () => {
         const result = parseHeadersInput('extraHeaders');
         expect(Array.from(result)).toHaveLength(0);
     });
-})
-
-describe('parseResponse', () => {
-    // https://www.vaultproject.io/api/secret/kv/kv-v1.html#sample-response
-    it('parses K/V version 1 response', () => {
-        const response = JSON.stringify({
-            data: {
-                foo: 'bar'
-            }
-        })
-        const output = parseResponse(response, 1);
-
-        expect(output).toEqual({
-            foo: 'bar'
-        });
-    });
-
-    // https://www.vaultproject.io/api/secret/kv/kv-v2.html#read-secret-version
-    it('parses K/V version 2 response', () => {
-        const response = JSON.stringify({
-            data: {
-                data: {
-                    foo: 'bar'
-                }
-            }
-        })
-        const output = parseResponse(response, 2);
-
-        expect(output).toEqual({
-            foo: 'bar'
-        });
-    });
 });
 
-
 describe('exportSecrets', () => {
     beforeEach(() => {
         jest.resetAllMocks();
@@ -224,6 +190,18 @@ describe('exportSecrets', () => {
         expect(core.setOutput).toBeCalledWith('key', '1');
     });
 
+    it('intl secret retrieval', async () => {
+        mockInput('测试 测试');
+        mockVaultData({
+            测试: 1
+        });
+
+        await exportSecrets();
+
+        expect(core.exportVariable).toBeCalledWith('测试', '1');
+        expect(core.setOutput).toBeCalledWith('测试', '1');
+    });
+
     it('mapped secret retrieval', async () => {
         mockInput('test key|TEST_NAME');
         mockVaultData({
@@ -267,4 +245,16 @@ describe('exportSecrets', () => {
         expect(core.exportVariable).toBeCalledWith('KEY', '1');
         expect(core.setOutput).toBeCalledWith('key', '1');
     });
+
+    it('nested secret retrieval', async () => {
+        mockInput('test key.value');
+        mockVaultData({
+            key: { value: 1 }
+        });
+
+        await exportSecrets();
+
+        expect(core.exportVariable).toBeCalledWith('KEY__VALUE', '1');
+        expect(core.setOutput).toBeCalledWith('key__value', '1');
+    });
 });

src/entry.js

@@ -0,0 +1,10 @@
+const core = require('@actions/core');
+const { exportSecrets } = require('./action');
+
+(async () => {
+    try {
+        await core.group('Get Vault Secrets', exportSecrets);
+    } catch (error) {
+        core.setFailed(error.message);
+    }
+})();

src/index.js

@@ -1,10 +1,7 @@
-const core = require('@actions/core');
+const auth = require('./auth');
-const { exportSecrets } = require('./action');
+const secrets = require('./secrets');
 
-(async () => {
+module.exports = {
-    try {
+    auth,
-        await core.group('Get Vault Secrets', exportSecrets);
+    secrets
-    } catch (error) {
+};
-        core.setFailed(error.message);
-    }
-})();

src/secrets.js

@@ -0,0 +1,76 @@
+const jsonata = require("jsonata");
+
+
+/**
+ * @typedef {Object} SecretRequest
+ * @property {string} path
+ * @property {string} selector
+ */
+
+/**
+ * @template {SecretRequest} TRequest
+ * @typedef {Object} SecretResponse
+ * @property {TRequest} request
+ * @property {string} value
+ * @property {boolean} cachedResponse
+ */
+
+ /**
+  * @template TRequest
+  * @param {Array<TRequest>} secretRequests
+  * @param {import('got').Got} client
+  * @return {Promise<SecretResponse<TRequest>[]>}
+  */
+async function getSecrets(secretRequests, client) {
+    const responseCache = new Map();
+    const results = [];
+    for (const secretRequest of secretRequests) {
+        const { path, selector } = secretRequest;
+
+        const requestPath = `v1${path}`;
+        let body;
+        let cachedResponse = false;
+        if (responseCache.has(requestPath)) {
+            body = responseCache.get(requestPath);
+            cachedResponse = true;
+        } else {
+            const result = await client.get(requestPath);
+            body = result.body;
+            responseCache.set(requestPath, body);
+        }
+
+        const value = selectData(JSON.parse(body), selector);
+        results.push({
+            request: secretRequest,
+            value,
+            cachedResponse
+        });
+    }
+    return results;
+}
+
+/**
+ * Uses a Jsonata selector retrieve a bit of data from the result
+ * @param {object} data 
+ * @param {string} selector 
+ */
+function selectData(data, selector) {
+    const ata = jsonata(selector);
+    let result = JSON.stringify(ata.evaluate(data));
+    // Compat for custom engines
+    if (!result && ata.ast().type === "path" && ata.ast()['steps'].length === 1 && selector !== 'data' && 'data' in data) {
+        result = JSON.stringify(jsonata(`data.${selector}`).evaluate(data));
+    } else if (!result) {
+        throw Error(`Unable to retrieve result for ${selector}. No match data was found. Double check your Key or Selector.`);
+    }
+
+    if (result.startsWith(`"`)) {
+        result = result.substring(1, result.length - 1);
+    }
+    return result;
+}
+
+module.exports = {
+    getSecrets,
+    selectData
+}