When starting from scratch first build a set of scripts (bash and golang) which start with a git pull to get the fresh code for a given branch then the script issues all the command line steps you can do yourself to compile the code into executable bundles.  This includes issuing Docker commands to cut fresh docker images for the various microservices.  Then a script launches all the various Docker containers.  Now the app can get fresh code pushed to domains where people can login and use the app from their client (mobile or browser) whenever a developer requests a domain needs a refresh.

The next step for us was to setup a github webhook server on each server for each domain which subscribes to all git push performed by our developers.  If a given git push matches the github code branch a domain is running the webhook would trigger launch of the above script to compile then deploy the app

All of above is just automation to replace what someone was originally manually issuing themselves at the command line.  As need appeared a new step was issued manually until it got crafted into above tooling.


copy contents of file on ansible server to another file on remote servers

Copy module should work

- hosts: all
  tasks:
    - copy:
        src: /data/new_config
        dest: /opt/zservice/etc/config
        owner: root
        group: wheel
        mode: '0750'
        backup: yes

If you worry about the number of the hosts take a look at serial.


How to create replicas in docker-compose when `scale` is deprecated?

@decoomanj on github indicated that scale has to be replaced by deploy in conjunction with replicas.

According to mrampant-nist on github one has to use --compatibility as well.

Let's test this. First of all all docker-compose.yml is needed:

version: '3.3'
services:
    nginx:
      image: nginx
      scale: 6

and issuing:

docker-compose up

results in:

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.nginx: 'scale'

Applying @decoomanj's suggestion, i.e. replacing replicas:

version: '3.3'
services:
    nginx:
      image: nginx
      deploy:
        replicas: 6

results in:

WARNING: Some services (nginx) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.

Running docker-compose --compatibility up starts 6 services:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
f19aa54634d7        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 4 seconds        80/tcp              6233_nginx_5
749cff4ac1c3        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 3 seconds        80/tcp              6233_nginx_3
b86ed7b217fe        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 1 second         80/tcp              6233_nginx_6
0b92e5ef4a0e        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 2 seconds        80/tcp              6233_nginx_2
e414bca52b53        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 4 seconds        80/tcp              6233_nginx_4
e9f33dc634a8        nginx               "nginx -g 'daemon of…"   6 seconds ago       Up 5 seconds        80/tcp              6233_nginx_1

In conclusion, replicas has to be replaced by deploy and replicas and --compatibility is needed to solve the issue.


Is there an elegant way to use a Dockerfile argument to tag an image?

You can use a Makefile for this:

PHPBB_VERSION := 3.2.4

build:
    docker build \
        --build-arg PHPBB_VERSION=${PHPBB_VERSION} \
        --tag phpbb:${PHPBB_VERSION} - < Dockerfile

Dockerfile:

FROM php:7.2-cli

ARG PHPBB_VERSION

RUN wget -nv -O- https://github.com/phpbb/phpbb/archive/release-${PHPBB_VERSION}.tar.gz | \
    tar xzf - --strip-components=1

Then, in a shell: make build.


Docker: subnet clients can't ping other subnet

The problem was in this line:

sudo docker network create --internal --subnet=192.168.2.0/24 netR

The --internal option makes the network unreachable from other Docker networks.

The solution was to remove this option.


Docker MongoDB image - How to specify credentials other than in the compose file?

Secrets support for mongo is built in. You can see in that image's entrypoint script that if you add _FILE to end of username/password values it will pull from those secret files.

As for .js files, if you're storing decrypted passwords in those files, you'd need to make a custom entrypoint script or something that will take envvars you're passing in for which secrets to use and run a jq or sed on the file to replace I guess. You can store the .js files themselves in a secret as longas its smaller than 500Kb (max size of a secret).


I think you're on the right track. Just need a tiny bit more logic on the backup process, then S3 can handle the rest.
Consider this:

A backup is uploaded every day.
If today is January 1st (for example), the backup is given a "yearly" tag.
Otherwise, if today is the 1st of the month, the backup is given a "monthly" tag.
Otherwise, if today is a Sunday, the backup is given a "weekly" tag.
Otherwise the backup is given a "daily" tag.

Next, set up S3 Bucket Lifecycle Policies with each tag as the filter.

Transition "yearly" backups to Glacier after 365 days (kept indefinitely)
Delete "monthly" backups after 365 days (1 year)
Delete "weekly" backups after 183 days (6 months)
Delete "daily" backups after 31 days (1 month)

References:

https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html
https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html


Can I run Chef/Puppet server from Azure and manage my on-premise infrastructure?

The prerequisites for a chef server are here and common to any installation.

You only need to allow https to port 443 from your inner infrastructure to the cloud server.

If you have a proxy with SSL interception I'd recommend adding this proxy certificate to each client cacert.pem and set an environment variable SSL_CERT_FILE=<chef_install_path>embedded/ssl/certs/cacert.pem so other call from different ruby tools using openssl will also work.


What is Ansible's config equivalent of `--vault-password-file`?

Here is the definition:

DEFAULT_VAULT_PASSWORD_FILE = get_config(p, DEFAULTS, 'vault_password_file', \
'ANSIBLE_VAULT_PASSWORD_FILE', None, value_type='path')

This means that you either put in ansible.cfg or playbook:

vault_password_file: ~/.vault_pass.txt

Or in your shell defined this variable:

export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt


All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...


How to attach two (or more) networks to a container so they are available when the container starts?

Do that:

You cannot attach several networks using the --network option. You have to rely on the docker network connect command. So if you want the multiple network interfaces to be available at container startup, you have to do it in three steps:

create the container,
connect the extra network needed,
and finally run the container.

# create a container, attaching it to one user-defined bridge:
docker create --name app3 \
              --network back-end \
              ${IMAGE}

# connect the other bridge networks:
docker network connect front-end app3

# start the container
docker start app3

docker exec -it app3 ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
465: eth1@if466: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:13:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.19.0.3/16 brd 172.19.255.255 scope global eth1
       valid_lft forever preferred_lft forever
467: eth0@if468: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:14:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.20.0.3/16 brd 172.20.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Dont' do that:

It is tempting to do that instead:

# You probably DON'T want this:
# create a container with the default network settings
docker create --name app3 \
              ${IMAGE}

# connect the user-defined bridge networks:
docker network connect back-end app3
docker network connect front-end app3

# start the container
docker start app3

However, there is a pitfall here: when creating the container without an explicit --network option, Docker attaches it to the default bridge. Then the two network connect commands will add two more networks. And the container end-up bein attached to the front_end and back_end networks as expected. But also to the default bridge, something you probably don't want when using user-defined networks.


How to send local files using Terraform Cloud as remote backend?

May be a better way to manage private key is using protected variables into the Terraform Cloud. Otherwise you have to save it in repo that is not secure.


Is it possible to design conditionally executed tasks in Bamboo?

Atlassian Labs have now provided this functionality as an add-on: https://marketplace.atlassian.com/apps/1219706/conditional-tasks-for-bamboo?hosting=server&tab=overview

Only available for versions 6.7.1+


Confluent Kafka on K8s - offset is stuck, lag increases, what can be done?

Things we have checked were what happens with the background io queue and the buffer. It turned out our producers were ok as lag was increasing, but as offset was static on check of consumer handlers it turned out we don't send acknowledgement to Kafka on read, leading to Kafka sending the same message after timeout on each attempt to consume. On java code side we have just made sure to send ack.


Ansible: Use inventory_hostname variable in lineinfile module

The issue seems resolved, If I escape the backslash in the regex too:

- name: Remove LE webroot definition
    lineinfile:
      path: "/etc/path/to/config/{{ inventory_hostname }}.conf"
      regexp: "^{{ inventory_hostname | replace('.', '\\.') }} = /path/to/a/directory"
      state: absent


What methods are available for connecting many-to-many containers?

The classic answer to balancing load is, of course, a load balancer. There are many different methods of doing this, but we're interested in ones that work well with dynamic containers.

This is one of the things an orchestration layer will address. In Kubernetes, for instance, you would create a Tensor Flow service, and scale it to however many pods you need (or use autoscaling!). Kubernetes then provides an ip for the service that kube-proxy load balances across the pods.

There are also service mesh options that replace this layer and perform a similar function.


Why does docker not restart on a stack trace?

The command line is order sensitive. This command:

docker run bot --restart-policy always

Runs the bot image as a container with the value of CMD set to --restart-policy always. To modify the restart policy of the container being run, you need to pass the option after run and before your image name:

docker run --restart-policy always bot


Why many companies prefer Jenkins than Aws CodePipeline?

Jenkins has been around for longer so it has a larger installed base.  It doesn't matter if AWS has something that is cheaper and better.  Many places don't want to put the time and energy into upgrading Jenkins.  Migrating to a new CI pipeline is even "scarier" -- regardless of whether it would be a wise strategic decision.

Jenkins 1.x made this far worse because of needing extensions for basic functionality.  Different departments wanting different plugins that wouldn't play well together led to multiple Jenkins instances.  Even if the technical challenge of merging these were relatively minor, the chance of getting departments that could never agree on anything before to suddenly decide that they can migrate to some new thing -- that is missing some feature from a plugin they can't live without -- seems pretty slim.


Why not use configuration management tool instead of Dockerfile?

The tool you need is Packer using Docker as the "builder" and Chef as the "provisioner". Then you can add the resulting image to your repo and reuse it without having to pack again, until your recipes change.


How to detect what interface has private IP in Ansible

Peter's module not working in CentOS and I write my own module:

#!/usr/bin/python
# encoding: utf-8

import subprocess
from ansible.module_utils.basic import *
from netaddr import IPNetwork

def get_priv_interface_info_for_ip(ip):
    network_config = subprocess.Popen(["/usr/sbin/ip", "-o", "addr", "show"], stdout=subprocess.PIPE)
    result = network_config.stdout.read()
    interface = ""
    network = ''
    ip_addr = ip
    for str in result.split("\n"):
        if 'inet ' in str:
            if ip in str:
                interface = str[3:str.find(" ", 3)]
                network = IPNetwork(str[str.find("net", 3) + 4:str.find("/", 3) + 3]).cidr

    return interface, ip_addr, network

def get_pub_interface_info_for_ip(ip):
    network_config = subprocess.Popen(["/usr/sbin/ip", "-o", "addr", "show"], stdout=subprocess.PIPE)
    result = network_config.stdout.read()
    interface = ''
    network = ''
    ip_addr = ''
    for str in result.split("\n"):
        if 'inet ' in str:
            if ip not in str and '127.0.0.1' not in str:
                print str
                interface = str[3:str.find(" ", 3)]
                network = IPNetwork(str[str.find("net", 3) + 4:str.find("/", 3) + 3]).cidr
                ip_addr = str[str.find("net", 3) + 4:str.find("/", 3)]

    return interface, ip_addr, network

def main(argv=None):
    if argv is None:
        argv = sys.argv

    fields = {"ip": {"required": True, "type": "str"}}
    module = AnsibleModule(argument_spec=fields)
    my_ip = module.params['ip']
    ifname_priv, ip_priv, network_priv = get_priv_interface_info_for_ip(my_ip)
    ifname_pub, ip_pub, network_pub = get_pub_interface_info_for_ip(my_ip)

    module.exit_json(changed=True, ifname_priv=ifname_priv, ip_priv=my_ip, network_priv=str(network_priv), ifname_pub=ifname_pub, ip_pub=ip_pub, network_pub=str(network_pub))

if __name__ == '__main__':
    main()

It works for servers with two network interfaces and provide information about public and private network interfaces.

Actual version of this module could be found in my GitHub.


Is it possible to securely delete a file from a virtual disk in a cloud provider?

You can't really use those tools with ext3 anyway. Wipe manpage has a large note about it:

  No secure deletion program that does filesystem-level calls can sanitize files on such filesystems, because sensitive data and metadata can be written to the journal, which cannot be readily accessed. Per-file secure deletion is better implemented in the operating system.

Moreover, the fact the data will be written really in-place is not guaranteed and you may be writing aside the original data.

That said the reason to do this goes back to a point where a data was written on disk on consecutive sectors on the same physical medium, usual storage nowadays use striping technology to spread data on multiple physical medium, each write in the system toward this storage will have an impact on all the stripes this file was part of. Each provider will have its own custom storage backend, but I'm pretty confident the same principles are used.

When you get the picture even higher, you're writing a file on a virtual hard disk which is itself a file on a storage pool, probably streched on multiple lun composed of raid array of physical disks.
There's so many IOs running on all those strates of striping/aggregation that the probability someone could recover a file by magnetic reading of physical disks (what wipe and shred should protect against) is very very very low as the storage used for this file has probably been reused a bunch of time already.

Even on existing ext3/ext4 filesytem, tools like extundelete have a hard time recovering any file if the disk is not unmounted/mounted read only very quickly after the deletion.


Parallel stages running on same agent in Jenkins declarative pipeline

I modified your Jenkins file as shown below.

pipeline {
agent { label 'agent01'}
stages {
    stage('build') {
        steps {
            echo "build"
            sh "ls -lrt && touch build.txt"
        }
    }
    stage('test') {
        parallel {
            stage('unit tests') {
                steps {
                    echo "unit tests"
                    sh "ls -lrt && touch ut.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
            stage('integration tests') {
                steps {
                    echo "integration tests"
                    sh "ls -lrt && touch it.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
        }
    }
    stage('deploy staging') {
        agent { label 'master' }
        steps {
            echo "deploy staging"
            sh "ls -lrt && touch dep.txt"
        }
    }
    stage('confirm release') {
        agent { label 'master' }
        steps {
            echo "RELEASE?"
            sh "ls -lrt && touch release.txt"
        }
    }
    stage('deploy production') {
        steps {
            echo "deploy production"
            sh "ls -lrt && touch produ.txt"
        }
    }
}
}

You need not stash and unstash files since it runs on same node and
    workspace.
Use a general node for all the stages except for the one that needs a user input.

The output is

    Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on agent01 in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] echo
build
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch build.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] parallel
[Pipeline]     [unit tests] { (Branch: unit tests)
[Pipeline]     [integration tests] { (Branch: integration tests)
[Pipeline]     [unit tests] stage
[Pipeline]     [unit tests] { (unit tests)
[Pipeline]     [integration tests] stage
[Pipeline]     [integration tests] { (integration tests)
[Pipeline]     [unit tests] echo
[unit tests] unit tests
[Pipeline]     [unit tests] sh
[unit tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [integration tests] echo
[integration tests] integration tests
[Pipeline]     [integration tests] sh
[unit tests] + ls -lrt
[unit tests] total 0
[unit tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[unit tests] + touch ut.txt
[integration tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [unit tests] sleep
[integration tests] + ls -lrt
[integration tests] total 0
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
[integration tests] + touch it.txt
[unit tests] Sleeping for 10 sec
[Pipeline]     [integration tests] sleep
[integration tests] Sleeping for 10 sec
[Pipeline]     [unit tests] }
[Pipeline]     [unit tests] // stage
[Pipeline]     [unit tests] }
[Pipeline]     [integration tests] }
[Pipeline]     [integration tests] // stage
[Pipeline]     [integration tests] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy staging)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
deploy staging
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch dep.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (confirm release)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
RELEASE?
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins 115 0 Oct 19 13:58 dep.txt
+ touch release.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy production)
[Pipeline] echo
deploy production
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 it.txt
+ touch produ.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS


What are the expected effects of the acquisition of GitHub by Microsoft?

Microsoft has heavily invested in their Git Virtual File System. It is unlikely then that such a key system central to Microsoft's internal development processes will be going anywhere any time soon, and GitHub is a smart dovetail with Microsoft's portfolio and strategic direction.

Microsoft has recently been moving towards more freemium services - which GitHub already is - and moving towards open source - the company was one of the top contributors to open source in 2017. Other notable examples are things like:

The Linux Subsystem for Windows
The open-sourcing of power shell
Contributions to the Linux Kernel

This also provides a strategic opportunity to integrate Visual Studio (which already has pretty good integration with Git) more tightly.

In short, Microsoft will probably do very little to harm GitHub - it will only work to make it solvent. This acquisition may have been very much about preserving GitHub to ensure that Microsoft will be able to continue to leverage its' features - the very same features you probably use. Nonetheless, guardianship of GitHub will be a difficult balancing act, as Microsoft must be careful not to alienate developers but they may have to end the gravy train as GitHub has been bleeding money for some time. Ultimately this acquisition is likely to be a good thing for the developer community as it will help to preserve and continue the public services that GitHub provides, but at worst things will look like the migration from SourceForge to whatever the new and shiny hot young thing is.


What are the daily job duties of a devops?

Keep learning other requirements for devops engineering. such as:

Operating System concepts (Process management, threads and concurrency, POSIX basics, I/O management, file systems, memory storage, ...)

as a devops engineer you need to know everything about the operating systems and how they work. to know everything about the operating systems, first you need to be a system administrator or sysadmin.

managing servers

server performance management helps in increasing agility and optimizes the performance of both physical and virtual servers.

service management

IT service management a cultural mindset to ensure that the desired outcome for the business is achieved. It incorporates principles and practices from various management approaches, such as lean manufacturing, organizational change management, system analysis and risk management.

bash scripting (vim, emacs, ...)

every sysadmin needs to have a good knowledge and experience in bash scripting.

working with terminals

every sysadmin needs to live entirely on the command line.

web servers (nginx, apache,...)

every sysadmin needs to know how servers work and how web servers send data.

CI/CD tools (jenkins, circle ci,...)

automation and automated testing improves agility and speed in scale.

infrascructure as code (docker, ansible, kubernetes,...)

IaC can be a key attribute of enabling best practices in devops.

cloud services (aws, azure, google cloud platform, heroku)

Here's the complete 2020 devops engineer roadmap: https://roadmap.sh/devops


Copying a file with special character & spaces

Note that the suspect filename character may not actually be ?, that could be just a filename displaying artifact if your system somehow doesn't properly display that character. BTW, R?glement sounds awfully close to the French Règlement.

One possible approach to accessing those files without worrying too much about typing in those characters or escaping them would be to try to wildcard them instead. In your case try something along these lines (use a more specific pattern if there are other files matching this one besides the 2 files of interest):

aws s3 sync R*glement*pdf s3://bucket_name/raw_files/ --sse


Docker Security Scanning: How to integrate Clair with a docker private registry?

Both Docker Registry and Docker Engine have API interfaces.
I consider it should be possible to implement a Docker plugin for this integration to have images scanned for example as somebody pushes them. (side note - Docker registry is an image registry, not container registry).

This scanner functionality is also possible through the commercial Docker Enterprise edition.
https://docs.docker.com/docker-cloud/builds/image-scan/

UPD here an example with clair/docker, looks quite simple
http://blog.xebia.com/docker-containers-vulnerability-scan-clair/


Approaching user adds and edits with Ansible

If editing critical flat-files across multiple servers in one place is too scary..

One approach I have often recommended is to let system users reside in /etc/passwd files, and to add an additional Name Service Switch (NSS) source for humans that need to log in. For big projects (lots of users authenticating against the PAM user data), that means LDAP, and for small ones (few can be hundreds of such users), nss_db + pam_userdb.

With multiple NSS and PAM sources for users, you can avoid disabling a server when the human user accounts distribution fails for whatever reason. Services (like your privileged Ansible account) should always depend ONLY on the standard /etc/passwd files. LDAP (or even MySQL) is probably overkill for some cases (like yours).

Consider the old BDB backed gems, designed originally (for big FTP file distribution servers) to provide faster logins than /etc files, but otherwise work nearly the same. They store the same passwd and groups flat files' content, but loaded into a (now non GPL) BerkeleyDB NOSQL key/value database file which is faster to read. For this purpose, it's just an extra place to store user/group authentication/authorization data.

NSS

https://en.wikipedia.org/wiki/Name_Service_Switch

https://www.systutorials.com/docs/linux/man/5-nsswitch.conf/

https://sourceforge.net/p/nssdb/home/Home/

PAM

https://en.wikipedia.org/wiki/Linux_PA

https://www.systutorials.com/docs/linux/man/8-pam_userdb/

https://www.cyberciti.biz/tips/centos-redhat-vsftpd-ftp-with-virtual-users.html

steps

collect your passwd, shadow, and group files on a secure server with
the nss_db Makefile
build the nss passwd and group db files using the provided nss_db
Makefile
extract the users and password hashes from shadow file and build PAM db file

awk -F: '{print $1;print$2}' shadow > users.txt # used like the vsftpd vusers.txt example

db_load -T -t hash -f users.txt pam-users.db

distribute the files to somewhere appropriate on your target servers
(maybe /var/db/authdb/*.db)
set up the target servers' pam.conf and nsswitch.conf files to use
these db files
Steps 1-4 are the repeatable build/deploy process.

why?

Once you have working pam/nss configs, your risk of breaking services
by updating/corrupting user auth/auth is very low.
User authentication will work on each server with no external network dependencies like LDAP or MySQL etc.
You can treat the DB files as immutable artifacts on the farm to reduce/prevent user auth snowflakes even if you can't have immutable servers.


According to How to let dpkg -i install dependencies for me? it sounds like dpkg doesn't have support for automatically installing a package's dependencies.

Personally I'd switch to a package manager which has such support, like apt-get. You'll like it ;)

Alternatively you could:

use a "helper" application around dpkg as suggested in the above-mentioned answer
manually collect the entire list of missing dependencies and install them as well. But it can be very tedious.

  I have the dependencies installed, but on the latest version

The version is part of the dependency, so technically the above means you don't actually have the dependencies installed. At least not for that particular version of OMG.

  and assuming that OMD and the packages that comes with it requires the
  older version?

Bingo. You have 2 options:

re-create an older version of your system which is compatible with your particular version of OMD. It could be based on Docker or some VM technology (or some plain old hw you may have lying around). I won't consider downgrading your system a solution :)
look for a newer version of OMD, hopefully one that's happy with what you have installed and on which its dependencies can be installed (some packages suport multiple versions installed on the same system). Also note from omdistro.org, you may be looking for another package:

  With the growing popularity of OMD, we came to a point were it makes
  sense to develop two specialized versions of OMD.

  Check_MK Raw: Check_MK Raw Edition focuses on Check_MK. Use this if you mainly use Check_MK. This is a smaller package including
  all components to run Check_MK.
  OMD-Labs: OMD-Labs contains all components of the original omd and some more addons like Grafana and Influxdb as well as Naemon and
  Icinga 2.


Is Serf an added value when a container orchestration platform like kubernetes is used?

Serf is gossip protocol implementation and has absolutely nothing to do with orchestration tools like k8s. Even if such protocol is used internally, then in most cases it isn't available for you applications running in given orchestrator.

So if your app or workflow depends on gossip protocol, then you will have use out of Serf, otherwise there is no point.

Gossip protocol is way of messaging to all nodes without:

Centralized point of knowledge
Single point of failure
Using whole network capability of any of the nodes

In simple words it works like "message chains" ("If you do not send this to N people you will never find your love" or something like that).

One node send notification to N out of M of its fellows
If you receive message that you have already received, then do nothing.
Otherwise go to step 1.

Given such simple protocol you can calculate probability when all your nodes will receive message with 99% chance.


Jenkins cannot reach nodes after upgrade

I found out via this StackOverflow question which pointed me to JENKINS-27624 which indicates that Java 8 is now required.

Update: I upgraded Java on all my nodes this morning and they are up and working.


Is Windows in general always slower than linux?

TL;DR depends on the use case with up to 12 times between best/worst

Comprehensive testing has been done over at Phoronix (28.03.18) and the results are very mixed. The tests looked at:

Clear Linux 21510
Debian 9.4
OpenSUSE 42.3 Leap
Ubuntu 16.04 LTS
Windows 10 Pro Build 16299
Debian 9 On WSL
OpenSUSE 42.3 On WSL
Ubuntu 16.04 LTS On WSL

The results are all over the show. In a few tests, the WSL (Windows Subsystem for Linux) is faster then native Windows (GraphicsMagick) and the Intel optimized distribution, Clear Linux, sometimes has the worst results (Netperf UDP request response). While in others Windows is twice as fast as all Linux distributions (CacheBench). All in all, deciding on performance is not conclusive. Choose the OS that provides the tooling you require.

Nonetheless, the article itself does provide a winner given that only first place finishes are taken into account:

  ... Clear Linux was the fastest of the operating systems tested with
  coming in first 40% of the time, Debian 9.4 in second with first place
  finishes 22.5% of the time, and Windows 10 Pro itself was the fastest
  12.5% of the time.

When only taking Java into account:

  In the Java tests, to no real surprise, the performance tended to be
  about the same across all tested operating systems.

And finally, in a compile test (Golang), which is the test that is most related to the use case in the question:

  The build performance on Windows 10 and WSL were slower than the bare
  metal Linux performance.


Geographically, where are the travis-ci jobs run?

Where are the travis-ci servers located? Does travis-ci leverage AWS EC2?

According to this documentation, the servers are located in the USA and AWS EC2 is used as well.

  Your code, depending on which platform or language runtime you're
  using, is run on virtualized servers running in:

  Amazon EC2 datacenters in Ashburn, VA, USA,
  MacStadium datacenters in Atlanta, GA, USA,
  Google Compute Engine datacenters in Berkeley County, SC and Council Bluffs, Iowa, USA.


Data sent from Logstash via elastic output plugin not showing in Kibana, but file output plugin works fine - what am I doing wrong?

Being unfamiliar with Kibana, I was not aware of the time constraint on data on the default search/display of just 15 minutes. The data coming in was timestamped (@timestamp key) via the 'date' plugin with the original open date, NOT the time of the, via TCP port to elastic, insertion event; thus, no data was showing and I had no idea that by default only the past 15 minutes of data based on @timestamp were displayed. If I had read the part about the time constraint I would have known. So I just adjusted the time to go back infinite years and saw my data.

So, if anyone else is having this problem, it's probably because you created a time-dependent index and have not clicked the 'time' button in the top right corner and changed the time frame.

This, my friends, is why you read the manual!


Is it possible to run every testing.xml file with different jenkins job?

I got the answer on StackOverflow :

You can pass the xml file name as parameter to the maven test command. First need to change the pom file as follows.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19.1</version>
        <configuration>
            <suiteXmlFiles>
                <suiteXmlFile>${SuiteXmlFile}</suiteXmlFile>
            </suiteXmlFiles>
        </configuration>
</plugin>

Then, you pass the different file name for each execution in goals and options like :

test -DSuiteXmlFile=mytestng1.xml

For second xml file :

test -DSuiteXmlFile=mytestng1.xml

You can configure each Jenkins job with different parameter.


A technique we've used in the past in similar situations is to get "management commitment" that imposes these rules to every team member:

Access to perform updates to the target deployment areas (i.e. production) is limited to selected automated systems, which have appropriate audit trails (= logging) of any kind of updates to the areas they manage.
Manual updates to the target deployment areas, for whatever reason, are no longer allowed by the typical team members (user ids) that used to be able (authorized) to perform these updates. Instead NEW (additional) user IDs will be created which will have all required permissions to (still) perform such manual updates, whenever needed. But to actually be able to use those new user IDs (= perform a logon with them), the team member who wants to perform a logon with such new user ID will have to perform "some" extra step to get access to the password for such new user Id. Ideally this extra step is automated also (use your own imagination how it should look like), but if anything else fails: just contact (= eMail, call, etc) the gate-keeper of the required password, including  "which issue they have to get fixed" (similar to your question).
Getting such gate-keeper in place is not an easy job. And the most resistance will come from ... the team members (for all sorts of reasons). Therefor a variation of those new user IDs (as in the previous step) is that each team member gets an extra user ID (with the password they decide themselves), but with an extra string attached to it: they are only allowed to perform a logon with that (extra) user ID if they actualy have a good reason to do so. And each time they perform such logon, they are required to file some type of report about "which issue they fixed" (similar to your question).

With these procedures in place, all that's left to do is to periodically review each of those reports / reasons why it was required to use such special user ID, and ask the question "Is there anything that can be done to further automate this, to further reduce the need for such special login?".

Update:

Quote from your extra comment below this answer:

  I think adding artificial barriers to fixing a deployment issue is counter-productive.

True it adds an extra barrier, but I'm not conviced it is "artificial". Because this is, to my knowledge, the only way to become aware of things those team members otherwise won't ever tell you, for reasons such as:

job security.
bad things/practises they prefer to keep hidden.
power they do not want to loose.


Why does Docker fail to start on AWS ECS instances. ECS stateful instance fails initialization. ECS stateful instance doesn't join the ECS cluster

The key here is the behavior of AWS AMIs in the context of ECS. By default the AWS ECS Amazon Linux AMI optimized for ECS is in charge of VOLUME management but there's a problem if you copy that AMI and try to launch a new instance -- the SECONDARY volume that ECS uses to deploy docker on can go missing.

Troubleshooting steps:

Check /var/log/cloud-init-output.log and see if you have an error message like:

  ERROR: /dev//dev/xvdcz is not a valid block device.

Check your instance mounted volumes and make sure that the Secondary Volume exists. Usually if you see the error above it means that only /dev/xvda is mounted.

FIX:

Enforce your Launch Configuration, Launch Template or script to always deploy the secondary volume /dev/xvdcz regardless of AMI configuration. Do not rely on the AMI as unfortunately the AMI flow has changed and might not preserve all the necessary configuration.


In kubernetes, using an nginx ingress controller what is recommended way of adding back end services?

Ingress controllers allow you to not manually wire stuff up, that's the whole point.

If you'd rather do it manually You could have an nginx array (either inside Kubernetes our outside of it) and just point to the ip/port of the services that expose your microservices. If you use an nginx array external to Kubernetes you'd have to use the externalIP service type. If you run the Nginx array on Kubernetes you could store your "manual proxy config" in a configmap so that all your nginx proxies get the same config.

I would definetely advise to not do this tho. Ingress controllers are built to simplify traffic routing, and as your complexity grows you will definetely want to take advantage of them. A good labeling scheme is absolutely paramount in Kubernetes, and that goes for ingresses aswell.


Determine the URL for a SCM trigger from inside the build

Some bad news and some good news :)

Bad news:

not a bug. this is a feature (default behaviour)

I don't understand why this bug is still open, either regression, or miscommunication.

Good news:

During setup of Global Shared Libraries jenkins allows to mark whether you should trigger user jobs upon changes in the library.
And, you can also mark this during library usage in the job's groovy.

Please refer to:

https://issues.jenkins-ci.org/browse/JENKINS-41497
https://stackoverflow.com/questions/46819344/jenkins-shared-groovy-library-git-merge-triggering-jobs


How to redirect the request to a different node when there is an unexpected exception and one of the pods gets restarted

You should configure a Readiness Probe in the Deployment for your app. When using a Readiness Probe, no traffic will be sent to instances that are not prepared to receive traffic.


When is it not appropriate to use serverless technology such as AWS Lambda?

There is likely no general answer to this question, but some reasons to not consider Lambda (or other serverless platforms) include:

App requires a language which is not available (C++, Haskell, Erlang, etc.)
Organization has existing substantial investment in containerized (or other) app development model, including developer knowledge, build and deployment pipelines
Portability is concern; i.e. the organization wants/needs to keep app in a state which is portable across platforms (AWS, Azure, various k8s implementations, and so on).
Cost. Do costs of running the Lambda function(s), as well as associated resources (API Gateway, VPC components) exceed budget? This is unlikely if the org. is already running in AWS, but can be a factor when compared against existing (and paid-off) on-premise resources.


Unable to start step function after account move

Issue is resolved by updating the subnet to a private subnet which uses a nat gateway instead of a public subnet with an internet gateway. This SO post lead my in the right direction.


Spark on a Docker swarm - workers won't accept jobs

TLDR; set workers' memory to 4G.

I reduced the test to the build-in Pi calculation:

./bin/run-example SparkPi 10

And got the same result; apparently 1G memory is insufficient even for simplest experiments.
Increasing both master and workers memory to 4G and the problematic behaviour was not reproducible. However before the systems start computation the message still could appear but just once, seems it is printed in a waiting loop.


Applying manifests in Azure DevOps pipeline

Once again, come across the answer right after asking the question.

Needed to look at the Kubectl task and not the Kubernetes manifest task. Particularly the Commands section:

- task: Kubernetes@1
  displayName: kubectl apply using arguments
  inputs:
    connectionType: Azure Resource Manager
    azureSubscriptionEndpoint: $(azureSubscriptionEndpoint)
    azureResourceGroup: $(azureResourceGroup)
    kubernetesCluster: $(kubernetesCluster)
    command: apply
    arguments: -f mhc-aks.yaml


Deployment slots share the same resources. The percentage of CPU used is what is taken by all slots combined.
After a swap, it should still monitor the production slot.


Jenkins Pipeline - Clean before checkout - git clean -fdx "failed to remove: Directory not empty"

We didn't find the cause, neither a real solution.

There is a workaround though:

Put a pipeline stage with rm -rf * before your scm checkout stage. This solves the problem but I will change the accepted answer to whoever posts the real reason and fix for the problem.

Sidenote:

Keep in mind that you will erase anything in your Jenkins workspace doing that. So be certain that you have everything you need in Git so it comes back with the checkout or get it back some other way.

If you rely on files in your Jenkins workspace from previous runs then this method is not for you!

(Although you might want to think about adapting your Job so files from previous runs are not needed anymore. Every run should contain the files it needs by itself and not rely on other runs.)


Is it possible to combine jinja templates in ansible like this is possible with erbs in puppet?

Yes, that is possible according to this post.

When a second jinja template was created and this was included in the base jinja it was called.

{% include "checks/subdue.j2" %}


Is it possible/fine to run Prometheus, Loki, Grafana outside of Kubernetes?

Yes--absolutely you can skip Kubernetes. Suggest that you still use Docker, but you could even just install the components separately.
I think it's easiest using Docker Compose (need to have package docker-compose installed on your host) to manage the containers.
Here is a docker-compose.yml that might be helpful:
version: '3'

services:
    prometheus:
        image: prom/prometheus:latest
        restart: unless-stopped
        ports:
            - "9090:9090"
        volumes:
            - /home/${USER}/opt/prometheus/data:/opt/prometheus/data:rw
            - /home/${USER}/opt/prometheus/config:/opt/prometheus/config:rw
        environment:
            SERVER_KEY: ${SERVER_KEY}
            SERVER_CERT: ${SERVER_CERT}

    grafana:
        image: grafana/grafana:latest
        restart: unless-stopped
        ports:
            - "3000:3000"
        volumes:
            - /home/${USER}/var/lib/grafana:/var/lib/grafana:rw
            - /home/${USER}/var/log/grafana:/var/log/grafana:rw
            - /home/${USER}/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:rw
            - /home/${USER}/etc/grafana/ldap.toml:/etc/grafana/ldap.toml:rw
        environment:
            SERVER_KEY: ${SERVER_KEY}
            SERVER_CERT: ${SERVER_CERT}
            GF_SECURITY_ADMIN_USER: ${GF_SECURITY_ADMIN_USER}
            GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD}
        depends_on:
            - "prometheus"

Can also use Docker Compose for Loki and Promtail.
Docker Compose reference: https://docs.docker.com/compose/compose-file


Jenkins and Kubernetes Design Involving GitHub

I would use the webhooks GitHub provides if you don't need something specific from Jenkins (Jenkins is in many cases a consumer of GitHub events/webhooks).
https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads#push
Using this webhook you could inspect each payload for specific parameters (ref, repo, etc...) and do something (like pull) if those parameters are met. In either case, you will need a service that can process the webhook/payload and perform the work/automation.


Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):

... allows an easy and fully customized build report display.
Features

Allow a rich summary report visible from both project and build page
Reports must be written in an XML format according to the syntax described in section Syntax Description
This plugin allow the parsing of several XML files
Reports displayed are ordered according to the XML file names
Element displayed are based on

In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.


Using Amazon VPC provided DNS trough AWS Site-to-Site VPN

Is it possible to query Amazon DNS server from our corporate network trough the VPN tunnel?

Directly? No. The recursive resolver available by default when you create a VPC can only be accessed by an IP address in that VPC. This is a restriction that Amazon imposes.

What you can do is create a forwarder using the Route 53 Resolver service or software like Unbound. You can then (conditionally) forward your onprem DNS for certain domains to it.

Some useful links:

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html#AmazonDNS

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html


Debug BackendConnectionErrors from Kubernetes Service LoadBalancer sesrvice

My solution to this problem was to rework my Services. The setup in my question had one K8s Service with ~10 ports. I reworked the setup to use one port per Service. The problem went away. I don't know why though. This makes me suspect something on the nodes themselves or some complexity in routing connections to the correct node port. I'm cautious of exposing too many ports again because of this.


This is a matter of implementation details and it's actually coming from the AWS Lambda product's implementation (which the doc you linked refers to), not from the Cloud Formation itself.

Presently the AWS Lambda implementation requires you to include the actual dependent packages in the deployment package (per the doc you referenced). Hence the CFT doesn't really need a way to specify such dependencies - it'd be useless as it can't be passed to Lambda.

Should Lambda infra end up supporting a way to just list the dependent packages, Cloud Formation might be enhanced to support specifying the list right in the template and passing it somehow to Lambda. But this is just speculation, it all depends on the AWS feature roadmap :)

Side note: the Google App Engine (GAE) example could be seen as a bit questionable as it's specific for just a small subset of that product's capabilities (applicable only to the GAE-supplied libraries for the python 2.7 standard environment). Maybe a better reference would be to the requirements.txt or package.json (or other language-specific) files used to specify external/third party package dependencies for GAE environments, as well as other Google Cloud Platform (GCP) products, including Cloud Functions (CFs) (which would be the GCP product closest matching AWS Lambda). See Specifying dependencies for CFs in Python, NodeJS 6 and NodeJS 8.


Is it possible to load an image into S3 via an URL?

Nope, credit goes to this SO answer, still valid today: How to upload files directly to Amazon S3 from a remote server?.

But if you have a server in the cloud, ideally in AWS, it'll probably go faster to execute the copy operations on that server - higher bandwidth and lower latency to/from such server when compared to your computer.


Download an artifact from Nexus to Jenkins

There are a few ways to do this, but by far the easiest is to put your credentials (either username/password or ssh keypair) into the Jenkins built-in credentials store and then use the sshagent step in your Pipeline script:

sshagent(credentials: ['my-credentials']) {
  sh('scp mynexus.com/my_artifact.zip ~/my_artifact.zip')
}

You may need to install the SSH Agent plugin to be able to use this step; I'm not sure if it's part of the default set of Pipeline plugins or not.


I'd have to say that I don't think it's possible and if you understood windows containers you really wouldn't want to do it anyway. Last time I checked, a windows container weighs in at gigabytes++ and that's huge to run notepad.
You can run xwindows apps in a container, but windows is, basically, complete shite in a container.
If you want a sandbox then you should checkout VirtualBox or Hyper-V. You can script either with Hashicorp Vagrant.


Install apache on awa ec2 from locally installed ansible server on my laptop

You need to put public ip address or hostname assigned by aws ec2 service, if your aws ec2 instance is public.

If your aws ec2 instance is located in vpc, you won't be able to connect it directly. Either you need to setup routing through internet gateway or you have to setup ansible server in vpc.


How to configure gitlab-runner to run in the same docker host as gitlab?

Management Summary

Set the pull_policy to "never" in the [runners.docker] section by calling:

docker exec -it gitlab-runner \
  vi /etc/gitlab-runner/config.toml

Detailed Description

It seemed the problem was a confusion with images and mounted volumes.

The documentation says to configure runners in:

/etc/gitlab-runner/config.toml

Which I did, but on the host machine.
When running gitlab-runner in a docker container, it mounts a volume:

... -v /srv/gitlab-runner/config:/etc/gitlab-runner ...

This means, that the configuration file in /etc/... in the container is mounted to /srv/... on the host machine.

In this file the pull_policy was not set, which defaults to always, what ultimately caused my problem in the first place.


Re-using Python virtual environment vs Re-building it

What are the general ways to tackle this problem?

The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.

Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:

python3.6 -m venv --copies venv
sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&\& pwd)"/' venv/bin/activate
sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*


What you are looking for is called dynamic provisioning of Jenkins' slaves, and yes this is doable. Setting it up depends upon your desired infrastructure provider/resource. The easiest example in my opinion is using AWS EC2 instances. Install the Amazon EC2 Plugin for Jenkins, and configure it with the appropriate access keys, IAM roles, etc. to allow it to provision EC2 instances in a specified VPC. Here is a short (but slightly outdated) guide.

If you are moving to K8's and Docker, you may want to consider looking into the Docker Plugin for Jenkins. If you are using bare bones infrastructure, this should allow you to more effectively utilize what current hardware you have.


You can achieve this using Git and Jenkins job. Create a Jenkins Job to deploy your code to server and Setup Git webhook to Call Jenkins job.

Check this link: https://www.serverkaka.com/2019/02/configure-github-webhook-in-jenkins-job-pipeline.html

  OR

Create an empty Git repo on the server
Write a Git Hook to deploy the code
Deploy from the local computer

Check this link: https://medium.com/@francoisromain/vps-deploy-with-git-fea605f1303b


How can integration tests for micro services be written?

Well, the name "integration" test means to test different pieces of the system together, I would not call a test that only uses one component an "integration" test.

You'd wire them up like in production (in a testing environment, of course), provide test data and let them do their thing. The components themselves do not need to know - they should not know - that they are under test.

The "wiring up" should be as similar to the production environment as possible, re-using configuration code where possible, to test those bits&pieces as well. Don't put them in the same repo just because you do integration tests.

To see whether the components work together correctly, you would have a test driver that exercises several of them together. It will set up the test data and look at the end results. If everything is green, then the APIs have, by definition, been used correctly. This means that there should be plenty of integration tests - if in doubt, I'd err on the side of having too many integration tests rather than too many unit tests. Obviously this depends a lot on what you are actually doing...


git reflog is showing plain text password used

You could use the git credentials.helper feature
https://git-scm.com/docs/gitcredentials


How to make a goreleaser script to build a deb file locally?

I have cloned your repo and try that in my local machine. Here is the steps :

Git clone
executing Dry run (testing everything before doing a release "for real" :

$ goreleaser release --skip-publish
show there is no error

 SIGNING ARTIFACTS
  • pipe skipped              error=artifact signing is disabled
• DOCKER IMAGES
  • pipe skipped              error=docker section is not configured
• PUBLISHING
  • pipe skipped              error=publishing is disabled
• release succeeded after 20.75s

execute goreleaser for release

$ goreleaser release
goreleaser will created dist folder inside project and this folder will consist of distribution packages (deb, rpm).

I have encounter some issues and here is what I do :

error=missing GITHUB_TOKEN, GITLAB_TOKEN and GITEA_TOKEN

create github or gitlab token ( https://github.com/settings/tokens) and put it as environment variabel

export GITHUB_TOKEN=xxxxyyyyyzzzzz

resolve the issue.

pre hook failed: xxxx is not within a known GOPATH/src

as I see in your goreleaser.yaml

hooks:
pre: dep ensure

you're using dep ensure, checking $GOPATH and make sure $GOPATH pointing to right path of your Go project.

error=dist is not empty, remove it before running goreleaser or use the --rm-dist flag

dist folder has been created before, you can either manually delete the folder or add flags --rm-dist when executing goreleaser command

$ goreleaser release --skip-publish --rm-dist

error=nfpm failed: rpmbuild not present in $PATH

this error occured as I was running on mac machine so there is no rpmbuild installed, installing rpm, rpmbuild solve the issue

$brew install rpm

error=git is currently in a dirty state, please check in your pipeline what can be changing the following files:
M Gopkg.lock

Goreleaser seems to check file diff, so as because running hook (dep ensure) updating the Gopkg.lock and this changes/updates are not pushed to git. The solution is always pushing the changes to git.

error=git tag v1.0.5 was not made against commit 3ae83eebd904d33cc549117254e857ebea04df90

reading from GoReleaser documentation which is "GoReleaser enforces semantic versioning and will error on non-compliant tags. Your tag should be a valid semantic version. If it is not, GoReleaser will error."

after pushing to git, make sure you have to update the tags, in this case I updates the tags to v1.0.6 (previously v1.0.5).

error=GitHub/GitLab/Gitea Releases: failed to publish artifacts POST http://xxxyyyzzz/releases: 404 Not Found []

make sure release text is there.


Port forwarding for Windows Server like SSH port forwarding

You can use a ssh reverse proxy from any terminal (that has the ssh bin) to communicate with your applications.

For instance try this on the client side:

 ssh -NL 3000:localhost:3000 root@10.10.10.1

This command won't print anything (except prompting for ssh key's password) and won't stop by itself. To kill the reverse proxy you can do ctrl + C

Now you should be able to curl your application the following way on the client:

curl 127.0.0.1:3000


How to get code onto aws ec2 instance using terraform and gitlabs

Setup a Jenkins server. Please do code checkout using Jenkins. Jenkins supports parameterized build. you can pass selected parameter from Jenkins UI to linux bash.
Then use linux's cli command to fire terraform.Terraform supports several workspace. You can use same code for dev, test and prod.
terraform workspace new dev
Use this command to create new workspace. After that , we can use this workspace name to your code. Below is the content of local.tf
locals {
  env = terraform.workspace
}

Like below use the same code for multiple env.
resource "aws_autoscaling_group" "asg" {
  name = "${local.env}-${local.project}-demo-asg"
}

Moreover to save your state file securely , you can use s3 as backend.
terraform {
  backend "s3" {
    bucket = "tfstate-demo"
    key = "keyfiles"
    region = "ap-southeast-1"
    dynamodb_table = "tfstate-demo"
  }
}

To separate backend and frontend service, use best practice of AWS. Create a public ALB for frontend and then move traffic to backend services via private ALB.
You can use Ansible as a configuration manager to build your server then create a AMI of that using packer. Later on, use the same AMI in ASG.


AWS ELB Application Load balancer, SSL not working

I solved it, I had forgotten that I had to add port 443 to the security group that my load balancer group belongs to.


Proper mapping for service architectures, Docker and BitBucket mapping - how do you do it?

One could also create libraries and call them using a dependency manager like a package.json if node is used or ivy if the code is written in java.


How to create dependency between modules in terraform

What you've described here is the recommended way to pass dependencies between modules, but it relies on features introduced in Terraform 0.12. You appear to be using Terraform 0.11, where unfortunately these features are not available.

For Terraform 0.12 it would be idiomatic to write it using the "first-class expressions" syntax, like this (with these examples spanning the three modules just like you did in your opening example):

output "alb_arn" {
  value = aws_alb.model-server.arn
}

module "common" {
  source = "./modules/common"
}

module "other" {
  source = "./modules/other"

  target_group_depends_on = module.common.alb_arn
}

variable "target_group_depends_on" {
  type    = any # only the dependencies matter, not the value
  default = null
}

resource "aws_alb_target_group" "modelServer" {
  # ...

  depends_on = [var.target_group_depends_on]
}

The main missing feature for the above in Terraform 0.11 is that it only supports resources in depends_on, and not other kinds of object like variables.

However, it's possible to work around that by introducing an additional do-nothing resource that exists only to be a dependency:

output "alb_arn" {
  value = "${aws_alb.model-server.arn}"
}

module "common" {
  source = "./modules/common"
}

module "other" {
  source = "./modules/other"

  target_group_depends_on = "${module.common.alb_arn}"
}

variable "target_group_depends_on" {
  type    = "string" # (because "any" isn't supported in 0.11)
  default = ""
}

resource "null_resource" "target_group_depends_on" {
  triggers = {
    # The reference to the variable here creates an implicit
    # dependency on the variable.
    dependency = "${var.target_group_depends_on}"
  }
}

resource "aws_alb_target_group" "modelServer" {
  # ...

  # Marking the null_resource as an explicit dependency
  # means this indirectly depends on everything the
  # null_resource depends on.
  depends_on = ["null_resource.target_group_depends_on"]
}

Note that in Terraform 0.11 the depends_on argument takes a list of literal strings containing resource addresses, not interpolated values. That was the specific reason for the error message you saw when you tried your example, but a reference like "var.dependency" would not have worked there either because only resource addresses are accepted as explicit dependencies in Terraform 0.11.


How to perform low-level inspecting of a Docker network traffic?

Turns out the answer is iptables, by adding a simple rule:

sudo iptables -I FORWARD -s <source ip> -j TEE --gateway <destination ip>

e.g.:

sudo iptables -I FORWARD -s 172.19.0.3 -j TEE --gateway 172.19.0.4

I can clone the traffic, so if I do this:

sudo iptables -I FORWARD -s <Container1 IP> -j TEE --gateway <Wireshark Container IP>
sudo iptables -I FORWARD -s <Container2 IP> -j TEE --gateway <Wireshark Container IP>

I will clone all traffic going between Containers1 and 2 to Wireshark container.

(Note that although this solution is a proof of concept, at the time of writing this answer)


How to Remove a Kubernetes (GKE) Cluster node-pool without down-time

If you cordon and drain the nodepool before deleting it, then you can avoid downtime. I use the following script (shamelessly taken from the lazyweb elsewhere and adapted to my needs):

oldpool=pool-1
oldnodes=$(k get no --selector='cloud.google.com/gke-nodepool='$oldpool -o json | jq .items[].metadata.name -r | xargs)

kubectl cordon --selector='cloud.google.com/gke-nodepool='$oldpool

for n in $oldnodes; do
  echo "draining $n"
  kubectl drain --delete-local-data --ignore-daemonsets $n
  echo "----------------------------"
done;


Unable to understand the logic behind this terraform function

It seems like the intent of this expression is to take the result of encoding that data structure which contains numbers and boolean values in quotes and produce a new version with those values unquoted.

For example, anywhere the result contains the sequence "true" it would be replaced with true, quotes removed to produce a real JSON boolean value rather than a string.

A more direct way to achieve that result would be to set the values in cadvisor_container_definition to be of the intended type immediately:

locals {
  cadvisor_container_definition = {
    name         = local.cadvisor_container
    image        = "google/cadvisor:latest"
    portMappings = [
      {
        containerPort = 8080
        hostPort      = 8080
      },
    ]
    mountPoints = [
      {
        sourceVolume  = "root"
        containerPath = "/rootfs"
        readOnly      = true
      },
      {
        sourceVolume  = "var_run"
        containerPath = "/var/run"
        readOnly      = false
      },
      {
        sourceVolume  = "sys"
        containerPath = "/sys"
        readOnly      = true
      },
      {
        sourceVolume  = "var_lib_docker"
        containerPath = "/var/lib/docker"
        readOnly      = true
      },
      {
        sourceVolume  = "cgroup",
        containerPath = "/sys/fs/cgroup"
        readOnly      = true
      }
    ]
    memory            = var.cadvisor_memory[var.env]
    memoryReservation = var.cadvisor_memory_reservation[var.env]
  }
}

I changed the literal values in the above to their numeric or boolean equivalents. I can't see from what you shared whether memory and memoryReservation are also supposed to be numbers, but if they are then that can be arranged by writing tonumber(var.cadvisor_memory[var.env]) and tonumber(var.cadvisor_memory_reservation[var.env]).

The above examples use Terraform 0.12 syntax. It's possible that the configuration you are looking at here is written for Terraform 0.11 instead, which would explain this unusual string substitution technique because the Terraform 0.11 version of jsonencode suffered from Terraform 0.11's tendency to convert all primitive-typed values to strings. If you are still using Terraform 0.11 then the strange expression you found is likely still needed, but if you are using Terraform 0.12 then a change like I described above should work to simplify the confusing expression to just a straightforward jsonencode:

jsonencode([local.cadvisor_container_definition])


TL;DR: Infrastructure as Code is a way to automate and backup your environment. In ideal case, after a disaster, you could restore your Infrastructure fully and automatically by Provisioning new resources, Restoring Configuration from Code Repository and Recovering Data from Backup.

Overview

Infrastructure as Code relies on three main concepts:

Automation of Configuration Management, called Continuous Configuration Automation a field pioneered by prof. Mark Burgess
Code Repository for changes to Infrastructure, where changes are first committed, documented, reviewed, tested and then deployed through automation.
Managed Infrastructure Provisioning. Infrastructure as a Service (IaaS). Either through Cloud Computing, Private or Managed Cloud or Managed Datacenter Services

Automation

Configuration Management is in its 3rd generation of tools. Building on CFEngine a new set of tools for Automated Configuration Management is being widely deployed now. The most popular in alphabetical order are Ansible, CFEngine, Chef, Puppet, PowerShell DSC and SaltStack. Each will have a Language to describe state of your infrastructure, Code Modules to apply those changes and provide ability to extend the tools, some Agent to execute those on the servers and a Central Repository of information.

They will generally operate in push or pull mode, either connecting to servers from a central location(s) and executing changes remotely or running on each server and pulling information about state from central location and that either in client/server model or in a distributed way.

The important concept is for the system administrator or site reliability engineer to not make changes directly to the infrastructure, but let the automation do changes. Anything done manually by a human should be considered either perishable, being soon corrected back by automation or in stricter form violating the integrity of the infrastructure and triggering destruction and rebuild of the affected components.

Code Repository

Code Repository, ideally separate from Repository holding Software, would be used to manage all changes to the Infrastructure and related Automation. It should hold Configuration files and templates, Playbooks (Cookbooks) describing process of changes to be reviewed, Code extending the CM automation tools, Provisioning configurations, Infrastructure Tests and Alerts, Staging/Deployment Tests, Documentation, Manual (not yet automated) Process Descriptions.

The important concept is to institute peer reviews for changes, to have record of all changes and ability to automatically revert to previous state in case of unpredictable and/or untested issues, ability to deploy to staging environment and test configuration changes and ability to automatically deploy changes without variation caused by human error.

Managed Infrastructure

Managing Physical Infrastructure is a real world task that goes beyond software and requires very different set of skills. By being able to abstract this layer through Cloud Computing or a Managed Datacenter, you have your team focus on the part of managing Infrastructure that adds business value.

While Cloud Computing offers a way to start and scale quickly at a later stage companies often realize some benefits and even significant savings in moving parts of the Infrastructure in their own data centers for a hybrid model. Owning or renting the hardware does not mean that you also have to employ the people who handle it. At this scale you need data centers geographically distributed around the world and having people with all the required skills in all places would be very expensive. Flying them around the world adds high latency to any changes and additional level of operational inefficiencies, which is another reason for outsourcing the datacenter management.

The important takeaway is that Managed Physical Infrastructure is often forgotten or overlooked concept, but just as important. Even if you've got everything automated, all configuration stored in a backed up code repository, unless you have a way to quickly provision, you have a huge bottleneck, which could easily erase all the benefits you've gained by the other two steps.


Docker compose to ECS, two services in one task

What you are looking for is a sidecar pattern where you run a container of your main application and a number of additional containers (one or more) in support of it. This is a well established pattern with orchestrators such as ECS but it's not possible to declare it with a docker compose syntax today. There is an open request talking about this. Can I ask you to +1 it and add your commentary/request there? It would help Docker to sense demand for this feature. Thanks!


Ansible hosts group reference does not work

I think it should be:
children:
    docker:
      hosts:
        01-dev:
        02-dev:

(The colon)


Error passing arguments to ansible playbook

I'd suggest that you first put some default values within the playbook (if you explicitly want your variables there, for the purpose of .. reducing interactivity/command line args lets say):

- hosts: all
  vars:
    user: greatuser
    passa: greatpassword

Second, I'm pretty sure that pass is a reserved word somewhere, although I'm failing to find it in the documentation. That's why I've changed your variable to passa. The final version of your playbook should look like this:

- hosts: all
  vars:
     user: greatuser
     passa: greatpassword

  tasks:
     - name: Stop services
       shell: "bash stopservices.sh"

     - name: Edit real hosts file
       shell: "bash editrealmachinehosts.sh"

     - name: Create dirs and cloning
       command: python3 {{ user }} {{ passa }}

Note the way used to reference to the username and password variables in the third task - {{ variable }} instead of variable.

Execute your playbook with the following command:

ansible-playbook createdeploy.yaml --extra-vars "user=ras passa=ras" -b --become-user=ras

As your target host is actually localhost (if I did understood you correctly), and since you're not showing your inventory file, I'd like to suggest that you might have to add -c local to the list of arguments of your ansible-playbook command.


In lack of a better idea I chose this rather long approach:

myid.j2

{% for host in groups['zookeeper'] %}
{% if host == inventory_hostname %}
{{ loop.index }}
{% endif %}
{% endfor %}

The idea is to iterate over all hosts and check if the current host being processed matches the host in the list of hosts. If it matches I write the iteration counter to the file.

When I ssh into the boxes I can see their myid files were created according to my need:

[root@quorumandmonitoring data]# cat myid
5


Why can I build this image from Docker for Mac but not on Docker CE for Ubuntu?

It is possible that the base image is different on each host. Can you confirm the images for php:5-fpm on each host are identical?

If the image from the Mac host is older, try updating it. You may need to update the package name in the dockerfile.

If you want to preserve the same behavior on your newer host, make sure you are pulling the image with the same hash or version-specific tag. It’s quite possible the newer version of the php image is on a newer version of Ubuntu. The package name may have changed between repos.


Is running load balanced autoscale apps cheaper on AWS Kubernetes than just using EC2 resources?

From your comment I assume a cpu load under 5% in average considering all instances together.

Each of your actual instance can be a pod on a kubernetes node, using quite larger instances you can reduce to a dozen of m3 or m4 machines with a single (or two) load balancer in front for the same quality of service.

You'll have machines used at 70% (ideally) and using an autoscaling group with a scaling policy to take care of spinning up a new node in case of burst.

The main idea is that you should be able to host on fewer machines with a better average load and thus reduce your cost, the first net gain is the reduction in number of load balancers, but this could be debated with Application Load balancers which allow to route by context or host.

Be warned there's a learning curve with kubernetes, but according to your description this sounds the perfect use case.

Beware of the 'all in k8s' trap, some things are better handled out of kubernetes, like large database systems or CPU/memory intensive task as they disrupt the load repartition and brings some more complexity. I.e: keep both possibilities, k8S and usual ASG/instances based hosting.


How do I use Jenkins to maintain versioning on a Python project?

If you follow Semantic Versioning you could achieve this as a step in the pipeline using Semantic Release.
This and the original tool it's based on are designed for automated releases as part of a continuous integration and delivery pipeline. From the docs:

The key point with using this package is to automate your releases and stop worrying about version numbers. Different approaches to automatic releases and publishing with the help of this package can be found below. Using a CI is the recommended approach.

In your case the workflow would be to do all the testing in pull request, and then when they are merged, a step in the develop branch would run semantic-release publish. Again, referencing the docs, this does the following actions:

Publish will do a sequence of things:

Run semantic-release version.
Push changes to git.
Run build_command and upload the created files to PyPI.
Run semantic-release changelog and post to your vcs provider.
Attach the files created by build_command to GitHub releases.

Part of the benefit of having this included in the pipeline is maintaining a clear and consistent changelog, with the help of a commit message linter like committizen.


How do I SSH into an EC2 instance from AWS gitlab-runner?

When you use public IP the traffic exits your VPC and comes back in, at which point it is no longer identified as coming from that security group.
So you can adjust your ssh connection and use private IP


Jenkins run build in docker on server (withDockerServer not working)

apparently docker needs to be installed on jenkins master in order for jenkins to be able to launch the docker on the server.

After installation of docker (even without the docker daemon running on jenkins master); jenkins was able to launch the docker on my-server.


They are the same thing with different names. It's a general term.


Are there any drawbacks of using a deb package as if it was a container to deploy an application?

First, while Docker is sometimes seen and used as a ad hoc packaging system, it actually solves a totally different problem: Docker is about running programs. The Docker system allows to describe services, that can be scaled at will and to control swarms of containers. Debian packages are for installing programs and they are able to handle dependencies between software versions.  Docker certainly don't qualify as a descent packaging system: each “package” can only have one dependency, the system has no “recursive build” option and does not support complex version constraints!

A possible answer would be that, if you are willing to write a Debian package for your application, you can also use Docker to deploy your application.  This can be achieved with a configuration script apt_setup.sh which would look like

apt-key add - <<EOF
-----BEGIN PGP PUBLIC KEY BLOCK-----
<YOUR RELEASE OFFICER PGP KEY GOES HERE>
EOF

cat >> /etc/apt/sources.list <<EOF
deb https://my.organisation.org/repo debian-jessie main
apt-get update -y
apt-get upgrade -y
EOF

and a Dockerfile along the lines of

ADD apt_setup.sh /root
RUN sh -ex /root/apt_setup.sh && rm /root/apt_setup.sh
RUN apt-get install -y my-node-js-package

(In your specific situation, the apt_setup.sh would be more complicated, adding the nodesource repositories and some helper packages such as apt-transport-https.)

It is therefore really possible to use Debian packages and Docker simultaneously, however …

  My gut […] is telling me that if deb packages were a good fit, it would be more common

This is a correct hitch which leads us to ask ourselves why Docker proves to be popular as a ad hoc packaging system, while it is not intended to be one. (See above.)

The “official” packaging system from a given distribution is just a possibility among many others to install software in some computing environment. There are many other sources available, like community-specific package managers such as npm or opam, port trees like pkgsrc and plain source code distribution.  From this perspective, it is easy to understand the success of Docker as an ad hoc packaging system:

Docker specifications are very close from a shell script and whatever source it comes from, we install software using the shell.
Docker has a “built-in” (paying) service for hosting artefacts it produces, the Docker Hub.

Now what are the strength of Debian packages over Docker images as a package system? The tight control over dependencies at installation. (The possibility to upgrade and downgrade also exists but has no practical importance if we are implementing the immutable-server pattern.) This leads to the

Conclusion

If you only have a single product deployed in a single version (which is typical for SaaS), your version management needs are very simple and using Docker as a ad hoc package manager should not have any hard drawbacks.  As soon as you work with several versions of a single product or several products, the complexity of the version constraints problem you need to solve increases and you need an appropriate tool for this, which might be Debian packages or some configuration management system if you are mixing software from different origins.


I did something similar with Akamai a few years ago that might work for you and I have a Plan B in case this doesn't fit your needs.

Plan A

assign each client a unique hostname.  So you might have client0001.example.com and so on.
optionally, you can let the client create a CNAME within their domain.  I was doing this for ad creatives so I wanted blame for content to go the originators of the content.  Since it looks to end users like they were serving it from their own domain and my org was just in the middle of a CNAME chain that most people wouldn't notice I'm guessing the grief went where it belonged.
then within the CDN have the host headers map to your client ID's
get regular reports from the CDN for hits/bandwidth used per client ID
bill accordingly

Plan B

If you don't want to do all of that mucking with DNS or if your provider won't do the sort of mapping described there's still a way to make this work:

give each client their own directory like /cli0001/, /cli0002/
write your own reporting that downloads the raw logs from the CDN provider and then pulls the client ID's out of each directory name and totals it all up for you.

Billing plans

There are different ways you can go about billing folks, but common choices are:

create tiers so that the first megabyte is $X/byte and the next gigabyte is a little cheaper per byte and so on
give everyone a quota for a fixed fee or call it "free" within a broader service.  Then charge for going over the quota at a reasonable rate.


Best practices to maintain forks of Git repository in an on-prem Azure DevOps Server

I think that the best way to answer your question is to take a step back to focus on your goal.  Based on the fact that you're using an on-premise Azure DevOps Server, I am assuming that you're working on a closed-source project (I apologize if this assumption is incorrect).  I believe that your goal is to control quality via PRs in this environment.
As this Atlassian tutorial states, "The Forking Workflow is most often seen in open source projects". The reason for this is that the fork mechanism was created for a very specific problem, "How do you give complete strangers the ability to contribute to a project without giving the world write permissions to your repository?".  Note, I occasionally use the forking mechanism for other fairly advanced purposes as well, but this is its primary use.  There is a fantastic book called Git for Teams that describes git workflows and their intents very nicely.
My first experience with git was working in GitLab on a closed-source project.  We were a team of ten employees working on an enterprise software application.  We were required by the process to fork the repo and create a PR to get our changes back. The result was a big learning curve for the engineers who were new to git and a lot of confusion around setting up remotes and getting the latest code.
Many people erroneously think that the only way to have a pull-request mechanism is to fork the code, probably because their only exposure is through PRs in the open-source community.  To quote Microsoft, "Pull requests can come from either topic branches or from a branch in a fork of the original repository".  A topic branch is just a fancy term for a bug/feature/etc branch.  When I am working with a closed-source group that is new to git, I will often disallow forks in the Project/Repository settings and I will enforce a branching policy of a minimum number of reviewers for a pull-request to the master branch.
Regarding best practices for branching strategies in git, the book I mentioned above is a great first step.  There are many standard branching strategies, a couple examples are:

gitflow which is great for teams that are new to git and has some awesome tools and many tools incorporate the gitflow pattern, like sourcetree.
trunk-based development a very lean development process.

Regarding your original question, I suspect that creating a new Azure DevOps project to store all of those unsightly extra fork repos would just add another level of complexity to your work.  You would need to enforce security for both projects, manage users and permissions for the new project, etc.  In this case subtraction is likely better than addition, I suggest removing forking and using a good branching strategy for your project (if possible). It would simplify your life significantly.
Please let me know if this did not answer your question or if you have any further questions.


Dockerfile COPY command with three input parameters

As per this docker documentation the copy instruction may have multiple sources but only one destination.
Your first copy is coping two file /etc/passwd and /etc/group to /etc.


Why screen folder in /var/run disappears in Docker container after running Ansible playbook?

Rather than solving the immediate problem you're having, let me suggest an alternative solution to your actual problem:

  I need screen, so I can use multiple interactive shells in one instance of Docker. If I run multiple Docker instances, each takes 1G of RAM, so 10+ instances will kill my RAM.

Run screen (or tmux, but screen if you insist) locally on your workstation.  Start up your docker container, and in the windows where you want to do something on the container, use docker exec -it <container-id> bash to get a new interactive shell on the container.  This will be much simpler than running screen inside the container, and allows you to also have windows running commands on your workstation.

You may also want to look into using Vagrant with the Docker provider to simplify the process of doing development on a Docker container.


Where do you store your configurations? Dockerfiles, gitlab-ci, application.properties, deploy scripts

If by "Shared" you mean that your team has access to the files then it should be stored in Source Control so that everyone on the team is using the same scripts.  You can consider a secrets manager so that team members do not share sensitive information (e.g., Docker Secrets, AWS Secrets Manager, Azure Key Vault).

If this will be publicly available you can still use a secrets manager to publish these files with the sensitive data removed.


How can I make use of bash functions in a makefile?

There are several options that can be used here. Maybe one of the easiest is to install each function as a real script in some directory that is added to the path.  Here is how to do this:

First, we choose a path name where to store all these functions, it can be a a directory in our project, where other utility scripts used in our Makefile live. We call it nvm_install_dir

Then we write a _nvm_trampoline script which use the name it is called as to trigger the right function, and create as many file aliases to _nvm_trampoline  as there are functions in the script.  Note the leading underscore, hinting at the “private” character of the script.

This script can go along the lines of

#!/bin/sh
. "${NVM_DIR}/nvm.sh"

if [ "${0##*/}" = '_nvm_driver' ]; then
    : NOP
else
    eval "${0##*/}" "$@"
fi

Here we use ${0##*/} to remove any path element from the name under which our script is called.  We install that script under ${nvm_install_dir} and run once the following utility script:

nvm_install_dir='SET-TO-ACTUAL-PARAMETER-VALUE!'
nvm_file="${NVM_DIR}/nvm.sh"
nvm_trampoline='_nvm_trampoline'

nvm_list_functions()
{
    awk -F'[(][)]' '$1 ~ /^nvm_[^ ]*$/{print($1)}' "${nvm_file}"
}

nvm_install()
{
    nvm_list_functions | {
        while read nvm_function; do
            ln\
                "${nvm_install_dir}/${nvm_driver}"\
                "${nvm_install_dir}/${nvm_function}"
            chmod 755 "${nvm_install_dir}/${nvm_function}"
        done
    }
}

nvm_install

It is a good practice to pack the nvm_install procedure in a real function instead of just inlining its body in the script, as it gives better testing options. (It is easier to comment out the call to nvm_install than the body of the function, if we want to experiment with the script.)

Ater this, the directory pointed to by nvm_install_dir is populated with aliases to _nvm_trampoline that are executable and delegate their work to the corresponding function. We only need to add this directory to our PATH when running make.

A second approach would be to generate pseudo commands for each nvm functions, with the following script:

nvm_file="${NVM_DIR}/nvm.sh"

nvm_list_functions()
{
    awk -F'[(][)]' '$1 ~ /^nvm_[^ ]*$/{print($1)}' "${nvm_file}"
}

nvm_generate()
{
    nvm_list_functions |  {
        while read nvm_function; do
            nvm_FUNCTION=$(printf '%s' "${nvm_function}" | tr '[a-z]' '[A-Z]')
            printf '%s=sh -c \047source "${NVM_DIR}/nvm.sh"; %s "$$@"\047 %s\n'\
                   "${nvm_FUNCTION}"\
                   "${nvm_function}"\
                   "${nvm_function}"
        done
    }
}

nvm_generate

The output of this program consists of make variable assignments

NVM_ECHO=sh -c 'source "${NVM_DIR}/nvm.sh"; nvm_echo "$$@"' nvm_echo
NVM_CD=sh -c 'source "${NVM_DIR}/nvm.sh"; nvm_cd "$$@"' nvm_cd
…
NVM_COMMAND_INFO=sh -c 'source "${NVM_DIR}/nvm.sh"; nvm_command_info "$$@"' nvm_command_info
…

It can be saved to a file Makefile.nvm or nvmtools.mk that can be included our Makefiles. Calling the function nvm_command_info is done with

${NVM_COMMAND_INFO} arg1 arg2 …

Using a customised bash profile is possible, using the assignment SHELL=/bin/bash --rcfile PATH-TO-CUSTOM-PROFILE -i but I would consider this esoteric enough to puzzle the maintenance programmer.


Gitflow with pull request or pull request

You can set up Branch Policies on your develop branch which requires a build to pass successfully before the Pull Request can be completed.

I have set this up for my team with great success; we very rarely will have a broken master branch. The PR does not run the main build, but instead runs a separate, slimmed down build that checks that the code compiles and the unit tests pass. The "Gated" build runs on Pull Request creation, the "Full" build is triggered on change in master branch (PR completion).


Expose kubernetes deployment on master's IP

I was wrong about the NodePort approach. It was firewall blocking the connection to my master on ports 30000-32767.

After unblocking this ports everything was working fine with NodePort service.


Include verbose output in terraform/terragrunt

You might be able to do something with the local-exec Provisioner to push the information out to an external script or file which you could monitor separately during the run.
The issue you might encounter is the run order, HCL isn't processed top to bottom like a "traditional" programming language, it works out a dependency tree for the resources it is creating, so might not run the local-exec when you'd expect them to be run.


Add stages to Jenkins declarative pipeline through shared library

Yes, but (to the best of my knowledge) you will have to use Scripted Pipeline within the shared library.  This means that the closure that you will pass to the shared library (in the below example, the closure is sh 'run-tests') will also have to be Scripted.
Jenkinsfile:
pipeline {
    agent any
    stages {
        stage('Test') {
            steps {
                runOnServers {
                    sh 'run-tests'
                }
            }
        }
        stage('Release') {
            echo 'release'
        }
    }
}

In your shared library, a file named vars/runOnServers.groovy:
def call(Closure closure) {
  parallel({
    SanJose: {
      node("test && san-jose") {
        stage('SanJose') {
          closure()
        }
      }
    },
    Dallas: {
      node('test dallas') {
        stage('Dallas') {
          closure()
        }
      }
    }
  }
}


Is there a way to run the Jenkinsfile from command line?

I've done this before. Essentially you can use the Jenkins CLI to import/export jobs, which will allow you to create and/or update a job with the definition you want to run. Be it JenkinsFile or some groovy script or whatever. Check this blog for details.

Once you (re)define the job, you will simply remotely execute it using the same Jenkins CLI. You can check the progress of the job from CLI and even read the logs to see progress. I cannot share the code, as the copyright is owned by one of my previous employers, but it is easy to recreate.


How to deploy applications to k8s using bitbucket pipelines?

You need to separate your daily authentication method of the CD authentication.

You should start understanding the different methods that K8S offers https://kubernetes.io/docs/admin/authentication/

On the other hand if you use some cloud solution like GCP you can benefit of the service account that the platform puts in all your instances, setting the right permissions to a service account allow you to to controll any cluster inside your project like in your local machine.


What is special about the admin in chef?

A Chef admin can create, modify and delete other users and clients. Basically an admin can manage access to Chef.


Turns out this was not a real issue. My apologies. At the end of that shell script, it did an echo which is what was returning the non-zero status code. After I took that out, it worked as expected.


Why is it necessary to have virtual machines to run multiple server instances?

Multiple apps on the same server

But what if I have a Node.js server application? I could run multiple of those on the same machine without any kind of virtualization.

Yes, you can run multiple apps or multiple instances of the same app on a server. The problem is that these apps will interfere.
Some examples of resources, that might interfere:

They would all like to listen to the same port e.g. 8080.
If they need more memory, they will eventually fight about the same memory.
If the apps is busy, they will fight about the same CPU resource.
They all would like to write logs to the same files on the file system.
They all might want to save data to the same place on the file system.

This needs to be managed in some way, and we don't want to do it manually.
With Virtual Machines on the same server, or Containers on the same server, each instance get its own IP-address, each instance can listen to what port it wants, e.g. all can listen to 8080. You can bound how much memory an instance has, e.g. 1GB - so if an app leaks memory, it does not affect the others. You can also bound how much CPU each instance can use, so they only use their fair share. Each instance has its own view of the filesystem, so all instances can write to the same path (as they see it) - even though it is technically different places.
Multiple instances - Availability

Why is even running multiple instances necessary?

What would you do when you need to patch the server and need to restart it? How will you upgrade your app to a new version? What happens if the hardware on your server crash? What happens when your single instance can not handle all the traffic?
The answer to all those questions, typically is: run more than one instance of your app, and let a load balancer forward the traffic to your instances that are alive.
Why virtualization?

Why were virtual machines needed?

They are not really needed. This is more about economics, a way to use your expensive hardware more efficiently.
If you start a new project, but a server - the server is very powerful and expensive. If you only start a Node.js app on this server, it will probably take long time before this Node.js app is so popular that it need to use the full server. It would be better if you could rent only a small part of the server in the beginning - that is how the Cloud with AWS and Google Cloud Platform works.


kubernetes allocating pods with the wrong IP

This was caused by missing --network-plugin=cni on the end of kubelet. Added that to the args and it all works now.


Tool to collect developer environment parameters

Instead of trying to watch, how about working together to define a scripted way of defining environments that is scm managed.  Having the configuration for deployment in the SCM insures they are always consistent and allows the collaboration and communication between team members about the ways they want to work.

There are many tools that can be used to script the configuration of the system level as well as the application level definitions.  Using this kind of definition based environment configuration not only helps with the development level, but then also helps with the deployment to later stages capabilities.

This would also facilitate the communication between team members a fundamental principle of DevOps.


Should AWS arn values be treated as secrets?

Well, from a security point of view, it's never bad to give people less information.

But, as long as you have sensible security policies in AWS, there is no reason that an arn has to be considered secret.

For example, if you know the arn of the role I use to access s3 from my EC2 instances, you can't do anything with that information unless I give you permission to assume that role from one of your roles.  Between accounts, this is a very explicit and bidirectional action; so it will not happen by accident/without you knowing.

Losing the credentials to an IAM user could let people assume roles and would be terrible, but them knowing the arn alone does nothing for them.


You shouldn't. Instead you should use proper secrets management system that would provide you unlocked SSL certificate.

By secret management I mean tooling like k8s secret management or Vault.

You can read Medium post about secrets.


Custom inventory file failing to parse properly

The keyword hosts is missing. Also colons : are needed after the hostnames. Fix the syntax, for example
all:
  children:
    control:
      hosts:
        moriarty.server.com:
        toby.server.com:
    managed:
      hosts:
        sherlock.server.com:

See How to build your inventory.


Jenkinsfile handling an expected job failure

One possibility would be to place the checks for all the necessary pre-conditions (i.e. including those bits of info that would need to be published) in a separate build step
to be executed prior to the actual build step.

If the checks in this preliminary step fail the build could be marked as aborted instead of failed. The aborted build would, hopefully, be a better indication that "nothing is actually wrong" (debatable, I know).

Marking the build as aborted could maybe be done in a manner similar to these SO answers:

Abort current build from pipeline in Jenkins
Jenkins - abort running build if new one is started

If those checks cannot be pulled in a step separated from the actual build step things are a bit more complicated:

the actual build step would have to somehow be configured to not fail the job. I know it's possible for steps performed in virtual environments using the ShiningPanda plugin, but other types of steps do not have such capability.
an additional step would have to be added after the actual build step. Its job would be to somehow determine if the actual build step failed and, if so, if failure was causing by one of those unfulfilled pre-requisite conditions (maybe by parsing the build logs, for example?), in which case it would abort the build. Otherwise it would pass or fail, depending on the actual build step outcome, to properly indicate the overall run status in Jenkins.