Techcle Wiki

QUICK START SINGLE POD

10 min read

Quick Start: k6 Single Pod Setup (No Multi-Pod Aggregation Complexity)

Status: Ready to implement
Timeline: 2-3 hours to first test
Confidence: ⭐⭐⭐⭐⭐ (mathematically sound, single source of truth)

Why Single Pod for Your Use Case?

Your load target: 10k-50k RPS (200-500 VU)

Pod ConfigurationMax RPS CapabilityGood For
256Mi/250m~5k RPSSmoke tests only
512Mi/500m~10k RPSLight to medium load
1Gi/1000m~25k RPSMedium to heavy load
2Gi/2000m~50k RPSHeavy load (peak)

Your 10k-50k RPS target: ✅ Single pod with 1-2Gi memory + 1-2 CPU cores handles it perfectly

Implementation: Single Pod k6 Operator Configuration

Current k6 Operator Setup (To Be Modified)

You currently have (assumed based on decision record):

# Current: Distributed across 3-5 pods
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: api-load-test
spec:
  parallelism: 5  # ❌ This causes aggregation issues
  script:
    configMap:
      name: test-script
  runner:
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
# NEW: Single large pod
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: api-load-test
spec:
  parallelism: 1  # ✅ Single pod, no aggregation needed
  script:
    configMap:
      name: test-script
  runner:
    # Larger resources for single pod
    resources:
      requests:
        cpu: 1000m           # 1 CPU
        memory: 1Gi          # 1GB RAM
      limits:
        cpu: 2000m           # Up to 2 CPU if needed
        memory: 2Gi          # Up to 2GB if needed

Key Changes:

  • parallelism: 1 (was 3-5 pods)
  • resources.requests.memory: 1Gi (was 512Mi)
  • resources.requests.cpu: 1000m (was 500m)
  • Limits are 2x requests for burst capacity

Example k6 Test Script (JavaScript)

Save this as test.js in your ConfigMap:

import http from 'k6/http';
import { check, sleep, group } from 'k6';
 
// Load configuration from environment or defaults
const BASE_URL = __ENV.BASE_URL || 'https://sandbox-sut.example.com';
const DURATION = __ENV.DURATION || '5m';
const VUS = parseInt(__ENV.VUS || '100');
const RAMP_UP = __ENV.RAMP_UP || '30s';
 
// Test configuration
export const options = {
  // Execution: Ramp up to VUS over RAMP_UP, then sustain for DURATION
  scenarios: {
    api_test: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: RAMP_UP, target: VUS },      // Ramp up
        { duration: DURATION, target: VUS },     // Sustained load
        { duration: RAMP_UP, target: 0 },        // Ramp down
      ],
      gracefulRampDown: '30s',
    },
  },
  
  // Thresholds: Pass/fail criteria
  thresholds: {
    // Response time thresholds
    'http_req_duration': ['p(95)<500', 'p(99)<1000'],  // 95th percentile < 500ms
    'http_req_duration{staticAsset:yes}': ['p(99)<100'],  // Static assets < 100ms
    
    // Error rate thresholds
    'http_req_failed': ['rate<0.01'],  // Less than 1% failure rate
    
    // Custom checks
    'checks': ['rate>0.95'],  // 95% of checks pass
  },
};
 
// Setup: Run once before test
export function setup() {
  console.log(`Starting test against ${BASE_URL}`);
  console.log(`Target: ${VUS} VUs, Duration: ${DURATION}`);
  return { startTime: new Date() };
}
 
// Main test function
export default function (data) {
  // Group 1: API Federation Endpoint
  group('Federation API', () => {
    let res = http.get(`${BASE_URL}/api/federation/endpoints`, {
      tags: { name: 'federation_endpoints', staticAsset: 'no' },
    });
    
    check(res, {
      'federation status is 200': (r) => r.status === 200,
      'federation response time < 500ms': (r) => r.timings.duration < 500,
      'federation has endpoints': (r) => r.json('endpoints').length > 0,
    });
    
    sleep(1);
  });
  
  // Group 2: Catalog Lookup
  group('Catalog Lookup', () => {
    let res = http.post(`${BASE_URL}/api/catalog/search`, 
      JSON.stringify({
        query: 'tax',
        limit: 10,
      }),
      {
        headers: { 'Content-Type': 'application/json' },
        tags: { name: 'catalog_search', staticAsset: 'no' },
      }
    );
    
    check(res, {
      'catalog status is 200': (r) => r.status === 200,
      'catalog response time < 300ms': (r) => r.timings.duration < 300,
      'catalog found results': (r) => r.json('results').length > 0,
    });
    
    sleep(1);
  });
  
  // Group 3: Auth/Token Request
  group('Authentication', () => {
    let res = http.post(`${BASE_URL}/auth/token`,
      JSON.stringify({
        client_id: __ENV.CLIENT_ID || 'test-client',
        client_secret: __ENV.CLIENT_SECRET || 'test-secret',
      }),
      {
        headers: { 'Content-Type': 'application/json' },
        tags: { name: 'auth_token', staticAsset: 'no' },
      }
    );
    
    check(res, {
      'auth status is 200': (r) => r.status === 200,
      'auth response time < 100ms': (r) => r.timings.duration < 100,
      'auth returns token': (r) => r.json('access_token') !== undefined,
    });
    
    sleep(1);
  });
}
 
// Teardown: Run once after test
export function teardown(data) {
  console.log(`Test completed. Started at: ${data.startTime}`);
}

Key Features:

  • ✅ Configurable via environment variables (BASE_URL, DURATION, VUS)
  • ✅ Ramp-up/ramp-down pattern (realistic load profile)
  • ✅ Multiple request types (GET, POST)
  • ✅ Checks/assertions on response status and content
  • ✅ Grouped requests for better organization
  • ✅ Thresholds to pass/fail the test

Kubernetes Manifest for Single Pod

Create k6-single-pod-testrun.yaml:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: api-load-test
  namespace: load-testing
  labels:
    project: hip
    environment: sandbox-sut
spec:
  # Single pod (no distribution)
  parallelism: 1
  
  # Test script
  script:
    configMap:
      name: test-script-v1
  
  # Runner pod configuration
  runner:
    # Resource requests: what pod needs to run
    resources:
      requests:
        cpu: 1000m              # 1 full CPU
        memory: 1Gi             # 1GB RAM
      limits:
        cpu: 2000m              # Can burst to 2 CPU
        memory: 2Gi             # Can burst to 2GB
    
    # Optional: environment variables for test
    env:
      - name: BASE_URL
        value: https://sandbox-sut.example.com
      - name: VUS
        value: "200"            # Starting VU count
      - name: DURATION
        value: 5m
      - name: RAMP_UP
        value: 30s
      - name: CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: k6-secrets
            key: client-id
      - name: CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            name: k6-secrets
            key: client-secret
    
    # Security context
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
    
    # Affinity: optionally pin to specific node
    affinity:
      nodeSelector:
        workload: load-testing  # If you have labeled test nodes
---
# ConfigMap with test script
apiVersion: v1
kind: ConfigMap
metadata:
  name: test-script-v1
  namespace: load-testing
data:
  test.js: |
    import http from 'k6/http';
    import { check, sleep, group } from 'k6';
    
    const BASE_URL = __ENV.BASE_URL || 'https://sandbox-sut.example.com';
    const VUS = parseInt(__ENV.VUS || '100');
    const DURATION = __ENV.DURATION || '5m';
    
    export const options = {
      scenarios: {
        api_test: {
          executor: 'ramping-vus',
          startVUs: 0,
          stages: [
            { duration: '30s', target: VUS },
            { duration: DURATION, target: VUS },
            { duration: '30s', target: 0 },
          ],
        },
      },
      thresholds: {
        'http_req_duration': ['p(95)<500', 'p(99)<1000'],
        'http_req_failed': ['rate<0.01'],
      },
    };
    
    export default function () {
      let res = http.get(`${BASE_URL}/api/status`);
      check(res, {
        'status is 200': (r) => r.status === 200,
      });
      sleep(1);
    }
---
# Secret for credentials (base64 encoded in real use)
apiVersion: v1
kind: Secret
metadata:
  name: k6-secrets
  namespace: load-testing
type: Opaque
stringData:
  client-id: "test-client"
  client-secret: "test-secret-123"

Collecting Results

Method 1: Pod Logs (Simple)

# Watch test in real-time
kubectl logs -f -l app.kubernetes.io/name=k6,k6_cr=api-load-test \
  -n load-testing
 
# Save results after completion
kubectl logs -l app.kubernetes.io/name=k6,k6_cr=api-load-test \
  -n load-testing > test-results.log

Output Example:

execution: local
script: /home/k6/test.js
output: -

scenarios: (100.00%) 1 scenario, 200 max VUs, 5m30s max duration

  ✓ auth status is 200
  ✓ auth response time < 100ms
  ✓ auth returns token
  ✓ catalog found results
  ✓ catalog response time < 300ms
  ✓ catalog status is 200
  ✓ federation endpoints
  ✓ federation response time < 500ms
  ✓ federation status is 200

checks.........................: 97.50% ✓ 4875  ✗ 125
data_received..................: 5.0 MB  16 kB/s
data_sent......................: 150 kB  500 B/s
http_req_blocked...............: avg=1.23ms min=0s    med=1.07ms max=12.34ms  p(90)=2.1ms  p(95)=2.5ms  p(99)=3.2ms
http_req_connecting............: avg=0.98ms min=0s    med=0.85ms max=8.23ms   p(90)=1.8ms  p(95)=2.1ms  p(99)=2.8ms
http_req_duration..............: avg=147ms  min=45ms  med=125ms  max=2.5s    p(90)=280ms  p(95)=380ms  p(99)=750ms
http_req_failed................: 2.50%   ✓ 125   ✗ 5000
http_req_receiving.............: avg=5.2ms min=0.3ms med=4.1ms max=125ms    p(90)=8.3ms  p(95)=12ms   p(99)=25ms
http_req_tls_handshaking.......: avg=0s    min=0s    med=0s    max=0s       p(90)=0s     p(95)=0s     p(99)=0s
http_req_waiting..............: avg=136ms min=40ms  med=115ms max=2.4s    p(90)=260ms  p(95)=350ms  p(99)=700ms
http_reqs......................: 5125   17.083333/s
iteration_duration.............: avg=3.15s min=3.04s med=3.12s max=6.28s   p(90)=3.18s  p(95)=3.22s  p(99)=3.35s
iterations.....................: 1025   3.416666/s

Confidence: ⭐⭐⭐⭐⭐ These numbers are 100% accurate (single pod, no aggregation)

Method 2: JSON Export (For Processing)

Modify test to export JSON:

# In k6 Operator spec, add:
spec:
  runner:
    args:
      - run
      - /home/k6/test.js
      - --out=json=/tmp/k6-results.json

Then collect results:

# Copy results file from pod
kubectl cp load-testing/api-load-test-xyz:/tmp/k6-results.json ./results.json
 
# Or if using emptyDir volume for results, extract differently
kubectl get pod -l app.kubernetes.io/name=k6,k6_cr=api-load-test -o name \
  | head -1 | xargs -I {} kubectl cp load-testing/{}:/results/output.json ./results.json

The JSON file contains all metrics in machine-readable format for further analysis.

Running Your First Test

Step 1: Deploy k6 Operator (If Not Already Done)

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
 
helm install k6-operator grafana/k6-operator \
  --namespace k6-system \
  --create-namespace \
  --set manager.image.tag=v0.50.0

Step 2: Create Namespace

kubectl create namespace load-testing
kubectl label namespace load-testing project=hip environment=sandbox-sut

Step 3: Create ConfigMap and Secret

# Create ConfigMap with test script
kubectl create configmap test-script-v1 \
  --from-file=test.js \
  --namespace load-testing
 
# Create Secret with credentials
kubectl create secret generic k6-secrets \
  --from-literal=client-id=test-client \
  --from-literal=client-secret=test-secret-123 \
  --namespace load-testing

Step 4: Apply TestRun

kubectl apply -f k6-single-pod-testrun.yaml

Step 5: Monitor Execution

# Watch the pod
kubectl get pods -n load-testing -w
 
# Stream logs
kubectl logs -f -l app.kubernetes.io/name=k6 -n load-testing
 
# Check TestRun status
kubectl get testrun -n load-testing -o wide

Step 6: Collect Results

# Save final results
kubectl logs -l app.kubernetes.io/name=k6 -n load-testing > test-results.log
 
# View summary
grep -A 30 "^execution:" test-results.log

Interpreting Results

What to Look For

✅ Good Signs:

  • http_req_failed: <1% (error rate < 1%)
  • http_req_duration p(95): < target threshold (e.g., <500ms)
  • checks: >95% pass rate
  • iterations: Matches expected (VU × duration)

⚠️ Warning Signs:

  • http_req_duration p(95): Growing over time = system degrading
  • http_req_failed: Rising trend = system under stress
  • iterations: Less than expected = timeouts happening

❌ Failure Signs:

  • Pod crashed (Out of Memory)
  • Connection refused errors
  • Very high p99 latencies (system struggling)

Validating Results Are Correct

Single-pod advantage: These metrics are guaranteed correct (no aggregation errors)

Compare to your expectations:

  1. Throughput: Expected RPS × duration = requests
    • Example: 100 VU × 5 min × (1 req/2 sec) ≈ 15,000 requests
  2. Response Time: Should be relatively stable during sustained phase
  3. Error Rate: Should be <1% (or 0% for well-behaved APIs)

Troubleshooting

Pod Crashes with OOM (Out of Memory)

Symptom: Pod killed after ~1 minute, 137 exit code

Solution: Increase memory:

resources:
  requests:
    memory: 2Gi
  limits:
    memory: 2Gi

Connection Refused Errors

Symptom: connection refused in logs repeatedly

Causes:

  • SUT is down
  • Network policy blocking access
  • SUT pod is restarting

Solution:

# Check SUT is reachable
kubectl exec -it <k6-pod> -n load-testing \
  -- curl https://sandbox-sut.example.com/api/status

Test Takes Longer Than Expected

Symptom: Test still running after specified duration

Cause: Ramp-down phase adds time

Solution: Reduce VU count or test duration if needed

Results Look Inconsistent

Symptom: Run same test twice, very different results

Cause:

  • Single pod is consistent (shouldn’t vary wildly)
  • Check if SUT is unstable
  • Verify no other load on system

Next Steps

After Validating Single Pod Works:

  1. Generate Reports: Use logs/JSON for documentation
  2. Baseline Metrics: Record “normal” performance
  3. Regression Testing: Use as CI/CD gate (commit causes 10% latency increase?)
  4. Capacity Planning: At what VU count does system degrade?

If You Need Multi-Pod Later:

  • This works up to ~50k RPS
  • If you need >50k: Then consider Option B (Gatling) or Option C (InfluxDB)
  • But you’ll have baseline confidence in metrics from single-pod approach

Files to Create

FilePurposeLocation
test.jsk6 test scriptConfigMap or file
k6-single-pod-testrun.yamlK8s manifestGit repo
.gitlab-ci.ymlGitLab trigger (optional)Git repo
test-results.logTest output (generated)Local

Confidence Checklist

Before declaring “success”, verify:

  • Pod started and ran to completion
  • Logs show expected checks passing
  • Response times are reasonable
  • Error rate <1%
  • Results are consistent across runs
  • You understand the metrics
  • No aggregation concerns (single pod = trusted)

Example: Complete First Run

# 1. Deploy k6 operator (one time)
helm install k6-operator grafana/k6-operator -n k6-system --create-namespace
 
# 2. Create namespace
kubectl create namespace load-testing
 
# 3. Copy test script and apply manifest
kubectl apply -f k6-single-pod-testrun.yaml
 
# 4. Wait for completion
kubectl get testrun api-load-test -n load-testing -w
 
# 5. View results
kubectl logs -l app.kubernetes.io/name=k6 -n load-testing | tail -50
 
# 6. Save for analysis
kubectl logs -l app.kubernetes.io/name=k6 -n load-testing > results-$(date +%s).log

Total Time: ~2-3 hours for setup + 10-15 minutes for test execution

Result: Baseline performance metrics with 100% confidence in accuracy ✅

Graph View