Mounted volumes can be used to preserve state between runs, apart from that Docker is stateless by design. To make a local directory available inside the Docker container, you can use the -v option, for example:

docker run -v /path/to/local/dir/:/mountpath/inside/docker ...

But if I understand your question correctly, you are asking for a faster feedback cycle while working on the Dockerfile. For that, I would recommend to start the container, and attach an interactive shell to try out your changes. Once you made some progress, update the Dockerfile.

$ docker ps
CONTAINER ID  IMAGE        COMMAND  CREATED           STATUS        PORTS  NAMES
f1217635695d  ubuntu:18.04 "bash"   23 seconds ago   Up 21 seconds         stoic_austin

$ docker exec -it f1217635695d bash
root@f1217635695d:/# ...

Also note that proper use of caching can speed up rebuilds of the Docker container after changes significantly.

  (from your question)
  1. Kill the container and prune it out of the system.

Pruning should not be necessary unless you run out of disk space. It should be sufficient to update the tag, for example:

docker build -t test .
docker run --rm -it test

test:latest will point to the last image id that was created. docker images will show that the previous images are not deleted, but as said, if you have enough disk space available, it is not a big concern.


How to find the total ips allowed by an AWS subnet?

AWS docs show what the reserved IP's in each subnet are: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html
The gist of it is, the first 4 IP's and the last IP of the subnet are off limits if your subnet starts with the same IP as the VPC.  It's probably a good rule of thumb to avoid assuming these IPs are going to be available in any subnet even though potentially the second subnet address reserved for DNS could be available for consumption in another subnet. A better rule of thumb to avoid under-sizing subnets in AWS. Like, never plan on using a subnet smaller than /24. You need them for auto-scaling groups, NAT gateways, VPN's, and networking hooks into other apps like DynamoDB or Lambda.


What is the difference between RUN and CMD in a Dockerfile?

Note: Don’t confuse RUN with CMD. RUN actually runs a command and commits the result; CMD does not execute anything at build time, but specifies the intended command for the image.

reference


Resources don't seem to show up in Terraform state after import

I figured it out. The stub resource has to have a provider that is configured with the proper AWS region. All these security groups were in us-west-2. You need this:
resource "aws_security_group" "rds-launch-wizard-2" { provider = aws.us_west_2 }

instead of this:
resource "aws_security_group" "rds-launch-wizard-2" {}

This assumes you have a provider already configured thus:
provider "aws" {
  region  = "us-west-2"
  alias   = "us_west_2"
}


'find' command suddenly not working in windows hosted agents

Find is a command in Windows that does very different thing to find in a posix shell.

And that would be the error you would get if you ran the Windows find (usually Windows find is located at C:\Windows\System32\find.exe


Gitlab runner fails with connection timeout

This is likely because your requests were dropped before they landed on the GitLab host. Check your firewalls and other security rules after assured that the 1234 port is open


Which option do I select for Security Group Source?

Use Custom and then start typing sg-... and it should pop up a list of available groups.


Why could the encrypted Ansible files not be decrypted although the vault password file has been set?

I found an issue with Ansible/Python causing a failure to decrypt.

Specifically, Ansible 2.7.10 and 2.7.13 under Python 3.6.8 on Ubuntu 18.04 would not decrypt (using ansible-vault edit -vvvvv) a vault file that was identical to file that decrypted OK on macOS 10.13 using Python 3.7.2 with same Ansible versions.

Upgrading to Python 3.7.3 on Ubuntu (using apt-get install python3.7 on 18.04) fixed this.

Your issue may be different, but if on Python 3.6 or lower, I would try upgrading to 3.7.


Your configuration refers to a file that is local on your system, but does not exist on the jenkins server.  The error message is trying to tell you this.  /c/Users/D35N/Downloads/GitHubProjeckt/Automation/GitHubProjeckt/newharbortestkey is not on the jenkins server.  You need to make this file available on the jenkins server by copying it up there, and it probably shouldn't be in your personal Downloads folder.
You may want to configure your script to look for a series of files and use the first one that exists.  This will let you have the file at different locations on your development box versus the server and it will still work in either.


Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the dashboards.

To change this in Grafana, open the Variables menu, then change $pod variable to refresh On Time Range Change:

Go back to the panels and change the time range (e.g., to "This Month"). The deleted pods for that period should be listed in the dropdown menu.


Can I run command from a program thats based on the root host from a containerized application?

This StackOverflow question has answers that could be useful for you, in particular, mounting parts of the hosts filesystem containing the host application.

But really, it doesn't seem to make much sense to run an application in a container and then try to directly access the host system. It seems to defeat many of the reasons to containerize an application in the first place.

It'd probably much better to put the required application in the container, along with Django. I see no reason why your container wouldn't be able to run docker itself (even though for some CI/CD use cases sibling containers are preferred over docker-in-docker approach).


Having trouble with building Docker image

Looks like the Dockerfile is not in the path, '.' means the current directory you are in.


What are best practices for over-committing resources in a staging environment?

Very good question; I'm interested to see what comes up here.

In terms of cost, obviously each company and team is different.  Ideally, everyone should be resource and cost conscious... but most people really don't seem to penny pinch by heavily monitoring and tuning their workloads (probably as that takes a certain skill level and tool set).

My Thoughts / Habits:

Unlike most on-prem environments, it is very easy to spawn up dynamic environments in the cloud (if you're automating properly).
So, when cost is prohibitive, you can just spin up full-scale staging environments in order to do load testing and then shut them down.
This will work for most use cases.  Obviously some things do only break when systems are alive a certain period of time under a certain load.
We usually keep identical but smaller environments in staging.  E.g. linearly scaled down presto clusters/spark clusters, k8s micro-services with less copies, and the like.
We try to keep the actual memory/cpu requests (or config) the same on any tool.  This allows for proper load testing without config changes for anything that scales.

Additional Thought

I'm pretty fond of the DevOps Toolkit book series by Victor Farcic.  If you're using kubernetes, the 2.5 book (https://www.devopstoolkitseries.com/posts/devops-25/) goes into detail on using prometheus and alert manager to detect over-sized or undersized resource requests, among other things.

If you're really into saving resources and testing well simultaneously, it may make most sense to implement and use this in staging and prod.  That way you can tune your alerting and automate your response so you can keep both environments sensibly sized for themselves in a consistent and enforced manner.

I think you'd still want to ensure you could set the staging environment (or another dynamic environment) to the prod settings to load test effectively though.


How to copy HTML files to include into nginx?

The docker build command builds Docker images from a Dockerfile and a “context”. A build’s context is the set of files located in the specified PATH or URL. The build process can refer to any of the files in the context. For example, your build can use a COPY instruction to reference a file in the context.
From docker build documentation.
As you run the docker build with the . path for the context, it will try to find the folder home/nicholas/html from the context root@mordor:/home/nicholas/docker/nginx
The COPY command documentation shows some examples


Kubernetes image concept compared to Docker?

k8s is a container orchestration tool
a dockerfile is used to build an image
k8s is able to run docker containers

One could use yaml files to deploy containers:

apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80


How to pass a /dev/disk device on macOS into Linux Docker?

Docker was built for linux, it runs on Mac through HyperkIT, which is a lightweight hypervisor. This means Docker only sees devices connected to the hyperkit hypervisor, rather than the devices connected to the Mac.

Unfortunately, Hyperkit has issues with USB device passthrough from your MacOS to the hypervisor (read more here) so the --device command won't be much use to you here. You have three options,

Option 1: Mount the ext4 drive to your Mac (not recommended)

Whilst your Mac doesn't natively support ext4, there are ways around this. I have installed ext4fuse on your Mac by following this tutorial. When you have mounted your device, add it to your ubuntu container with the -v flag,

docker run -v <Your_volume>:<dest_volume> -it ubuntu bash

Making sure that the directory of your volume has been added to the file sharing folders in Docker.

Option 2: create a docker-machine using the virtualbox driver and add the device to the virtualbox vm (still not recommended)

Follow this great tutorial here which goes through installing a docker machine with a virtualbox driver and mounting the USB stick.

Option 3: Run Docker on linux (recommended!)

I wouldn't recommend using Docker for Mac. It was built for linux and if you don't switch to linux you'll waste your time troubleshooting issues like these for no real reason.

I've got a mac, but run a few ubuntu cores in virtualbox and then install docker on top of them, that way I can play around with docker starm too (which you can't with docker Mac unless you use docker-machine!


This project is to support automation of running ansible playbooks. There is a GUI based project called Ansible Tower or AWX to do that, but ansible-runner seems to be the underlying layer of the automation in essence replacing ansible-playbook command.

There are some really serious issues for automation when trying to use ansible-playbook as part of some scripts. The output of ansible-playbook is the most awful thing under the sun when it comes to parsing. So you have to start with writing your own output plugin and then write a parser for that, then you need to figure out how to manage the environment variables and passwords and many other things. So ansible-runner is the answer to this. It would preserve the artifacts from running a task or playbook and make encapsulation of ansible in further automation possible.

You can run it from shell, you can run it as python module and there is even a reference container for it and obviously it is used by AWX. Further the project's event emitter will hopefully be much better pluggable to monitoring systems like ELK and to Message Bus type systems.


Dedicated/Separate Git Repo as Terraform Remote Backend

Using git for terraform remote state management is an interesting, but non-standard implementation. Storing binary / large files in git, integrating a locking system and a few other concerns come to mind. Using s3, Buckets or other storage options provide the multiple statefile management (versions in buckets, folders in buckets or terraform folder structure / workspaces) and other functionality you mentioned.
If you would like to pursue using git for remote state management, check out the community project terraform-backend-git. There is also a good discussion around it in the terraform issues section: https://github.com/hashicorp/terraform/issues/16616.
If you attempt to use the tool from the community, it would be great to hear about your experience. Good luck!


Using a ubuntu base image vs an application specific image (node, python, etc)

They're all functionally equivalent.  When you docker push or docker pull an image (or mention it in a Kubernetes manifest), all of its layers are pushed or pulled, so you always get a complete image with all of its dependencies at the point where it was built.  (It turns out the versioned ubuntu images are re-released pretty routinely to include current security patches.)

Building your application outside of Docker was pretty common before there were multi-stage builds.  There's nothing wrong with doing it, if it makes sense.  It makes more sense if your built application artifact is portable (in your example, Webpack JavaScript/HTML/CSS artifacts; Java .jar files work well here too).  This is also pretty easy to plug into an existing CI system that produces the artifacts, and then you just need to wrap them up in an image.

Using prebuilt images like nginx or node, or building your own on top of ubuntu or alpine, is mostly a matter of taste and organizational requirements.  Building an image FROM node will be easier than building your own Node image and can get you running faster; if you start FROM ubuntu then you know exactly who built it and what security updates and packages are in it.

If you're pinning to specific versions of software, the other important thing to be aware of is that the Docker Hub packages eventually stop getting updates.  If you're using node:8 that image isn't being automatically built any more; if you pin to a more specific version, node:12.13 or node:12.14.0 aren't getting updates either.  (As of this writing the Docker Hub node page lists 12.14.1 as the current LTS build.)  That could be an issue if you're forced to take a language runtime update to get an OS-level security fix.


Can I use AWS Step Functions to launch many different fargate 'runs'?

Yes, you can start ECS Fargate tasks from Step Functions in the way you've described.
Step Functions allows you to pass Parameters to the underlying service API.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html
[sidebar: that doc also specifies how to pass state inputs, which you've stated is a requirement]
For ECS Fargate, a single state machine with no unique configuration could look something like this:
 "States": {
    "${name}": {
      "Type": "Task",
      "Resource": "arn:aws:states:::ecs:runTask.sync",
      "Parameters": {
        "LaunchType": "FARGATE",
        "Cluster": "${cluster}",
        "TaskDefinition": "${task_definition}",
        "NetworkConfiguration": {
          "AwsvpcConfiguration": {
            "Subnets": ["${subnet_1}", "${subnet_2}"],
            "SecurityGroups": ["${security_group}"],
            "AssignPublicIp": "DISABLED"
          }
        }
      },

Now, you've asked about passing each task slightly different parameter values...
Thankfully, the ECS RunTask API has a helpful overrides: { containerOverrides: [] } parameter.
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html
Overriding the command (for task execution) or environment (for OS variables) should provide the customization you require without needing multiple ECS Task Definitions.
So, add each distinct task to your state machine, each ingesting state inputs and with its own unique override, and that should do it!


Is there any way to break long command into multi lines in GitlabCI

Since .gitlab-ci.yml is a Yaml file, then just use it's syntax.
For example, you may use >:
image: ubuntu
before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client rsync git -y )'
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo -e "$PRIVATE_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

SSH:
  tags:
    - mytag
  script: >
  - ssh ubuntu@host "which ssh-agent ||
    ( apt-get update -y && apt-get install openssh-client rsync git -y ) &&
    ssh ubuntu@192.168.1.2 "mkdir -p /home/ubuntu/test" &&
    rsync /home/ubuntu/test/.gitlab-ci.yml ubuntu@192.168.1.2:/home/ubuntu/test/.gitlab-ci.yml"

Sample usage of yaml multilines
There is a great resource about yaml multilines: YAML Multiline Strings
Also, there is a great SO answer about yaml multilines.
So let's speculate about pipeline based on these materials.
Since we have specific context (script for Gitlab CI) then we don't need all of theese 63 cases:
                      >     |            "     '     >-     >+     |-     |+
-------------------------|------|-----|-----|-----|------|------|------|------
Trailing spaces   | Kept | Kept |     |     |     | Kept | Kept | Kept | Kept
Single newline => | _    | \n   | _   | _   | _   | _    |  _   | \n   | \n
Double newline => | \n   | \n\n | \n  | \n  | \n  | \n   |  \n  | \n\n | \n\n
Final newline  => | \n   | \n   |     |     |     |      |  \n  |      | \n
Final dbl nl's => |      |      |     |     |     |      | Kept |      | Kept
In-line newlines  | No   | No   | No  | \n  | No  | No   | No   | No   | No
Spaceless newlines| No   | No   | No  | \   | No  | No   | No   | No   | No
Single quote      | '    | '    | '   | '   | ''  | '    | '    | '    | '
Double quote      | "    | "    | "   | \"  | "   | "    | "    | "    | "
Backslash         | \    | \    | \   | \\  | \   | \    | \    | \    | \
" #", ": "        | Ok   | Ok   | No  | Ok  | Ok  | Ok   | Ok   | Ok   | Ok
Can start on same | No   | No   | Yes | Yes | Yes | No   | No   | No   | No
line as key       |

Assumptions are:

we don't need trailing spaces at all
we don't need final newlines at all
we'll try to avoid quote escaping mess
Backslash is intended to combine multi-line scripts
almost doesn't matter how many spaces before the script

P.s. I'd suggest to lint your .gitlab-ci.yml using GitLab UI


What are some methods to measure the ROI for DevOps?

Great question! Most of us know investing in DevOps practices is a highly worthwhile pursuit for myriad reasons; we don't often justify DevOps on the impact to the bottom line alone, though.

Note: this is something of an opinionated question, and my answer is likewise, opinionated.

Tensibai wisely suggested we find the right metrics, and he used time-to-market as an example. This is a great big-picture approach.

As an alternative approach, my experience with bean-counters is that they don't need to--or necessarily want to--know the big-picture, they just want evidence of fiscal responsibility. They want to see a trend in the right direction.

Here are just a few fiscal wins:

calculate the server costs saved by leveraging auto-scaling in the cloud
for income-generating sites, extrapolate the cost-per-minute of downtime, then show improvements in MTTI and MTTR
again, for income-generating sites, estimate the cost-per-minute saved by leveraging highly-available architecture based on past incidents
if you've improved on your build and deploy pipeline, show that you've reduced regressions and errors in production caused by already-tracked faults
if you've made improvements to developer test environments, or even the tools and configuration on developer laptops, look at commit histories to see if new engineers are making their first contributions sooner after joining
perform a full cost comparison between cloud and on-prem, much like Gitlab did recently, to justify your infrastructure spending (a.k.a. savings!)

Showing that you're money-conscious and you have a few clear wins is often sufficient. I've certainly missed some obvious examples; feel free to add comments below.


OpenVPN - routing public domain over tunnel?

Yes, this is possible.

You can create a private DNS zone for example.io using AWS route53 in the AWS VPC where your openvpn server is running. Then you can point server.example.io to the private IP of the instance.
By default in AWS VPC, if a private DNS zone is present, it gets the preference. If the DNS record is not present, then it goes out of VPC for DNS resolution.

So, when your users are connected to the VPN, they all will be in the same network of AWS VPC. Hitting server.example.io will resolve to private IP and requests will be routed internally within VPN tunnel.

NOTE: If you are using multiple VPC or your server.example.io is in another VPC, you may need to add routes for that in your VPN configuration.


Ansible sporadic errors with Windows machines

You should probably create an Ansible issue for this as it's most likely a bug in Ansible.

The first error makes me think about WinRM pipelining:

Ansible 2.3.0 introduced an always-on WinRM pipelining feature (similar to SSH pipelining), and that may be behind this.
SSH pipelining can cause issues in Ansible for Linux, and it can be useful to turn it off, but this isn't yet possible for WinRM pipelining.

This related issue includes some Git commits that will re-enable 'non-pipelined' mode in a future release (should now be released in 2.4, possibly with a backport as part of 2.3.2 - see this comment)

Try upgrading to Ansible 2.4.1+ (which generally works well) to get the fix.  Or try downgrading to Ansible 2.2.3 to see if this helps - this will disable WinRM pipelining and may avoid other regression bugs in this area.

If you installed Ansible using pip, you can do  pip install ansible==2.4.1
to upgrade (or ansible==2.2.3 to downgrade), then if that doesn't help, do the same with 2.3.1 to upgrade again.
you should also upgrade to latest pywinrm as mentioned in the issues above


How to verify hashes of Ansible Galaxy dependencies?

The feature is currently (Nov 2019) not directly supported in Ansible Galaxy.

There have been related discussions in the past (#14604 and #1358), but the latest state can be followed in the new issue #2108.

In the meantime, the workaround described in the question can be used, at the cost of some verbosity.


Solution was pretty simple: you have to choose ARM architecture on the first step, AMI selection. Accustomed to choose x86 architecture, I didn't pay attention to that radio button. Also I didn't bother to hover over the forbidden icon on the first column in the image posted on question, which clearly stated what was the problem:

So solution is to select ARM on the previous step:


I believe some of the monitoring tools like Dynatrace and Scope show some neat stuff. This really depends on your platform (I'm assuming Kubernetes).

However, if you are running something else checkout Visceral. Here is a neat video of it in action. It's more complicated to get up and going.


Continuous Deployment is certainly a no brainer for mission critical applications but you may try Azure DevOps from MS which integrates with AWS beautifully and has provisions for environment approvals which can help move to that state in a user friendly and elegant manner. IMO, AWS code deploy is still not at par with Azure DevOps. And perhaps you may want to incrementally move there and refine your used case.
e.g. Instead of weekend deploy, you can make the process decoupled and efficient that you can do it on a Friday night, then setup blue/green in Beanstalk or EKS et to reach that state. Also for the 6000 hosts, you definitely want to try and see which can live in Kubernetes for ephemeral instances and review the infrastructure footprint. Don't forget cloud is expensive, a lift and shift of that many VMs would have exponential costs in cloud.
I work in healthcare so unsure about retailers, but thought I will suggest that tool (Azure DevOps) incase your client isn't familiar with DevOps for easier transition.
Also to be more comprehensive, consider monitoring, security etc also as a part of your effort in your white paper.
One of the most comprehensive DevOps roadmap (evolving) is this, you may consider all the aspects from this chart for your whitepaper -
https://roadmap.sh/devops


What functional differences are there between kubernetes, rancher and mesos?

First of all, Rancher actually contains implementations of both Kubernetes and Mesos within itself. However, they did make their own system called Cattle which is heavily based on Docker's Swarm. I'll touch upon this in the section for Rancher.

Secondly, since each one offers similar base features (load balancing, resource isolation, etc) I'll try to go more into what makes them different rather than focusing on differences among those common features unless they're significant.

Kubernetes

Highly performance focused, also featuring cloud storage orchestration (a feature missing from Mesos, although there's probably a plugin for it). Has API options to allow for automated scaling of resource usage as needed by individual containers (and for the reverse if a container is not being hit hard while others are).

Something important about Kubernetes is that unlike other container orchestration software it doesn't provide a comprehensive  configuration or any kind of comprehensive self-healing. Instead, it focuses on continuous deployment of multiple apps with an easy rollback system at the app level (as a result you might want to look into micro-services when using it).

Each app is a small piece and can be deployed/configured individually. They make a point in their docs to say that Kubernetes is not a traditional PaaS (platform as a service) system since it lacks middleware for virtual hardware or databases and it doesn't build your app itself. It's designed to (as they say themselves) remove the need for manual container orchestration, instead automating the process by continuously pressing towards a target app state.

Mesos

Monolithic in comparison to Kubernetes. Focuses on the big picture moreso than the individual services, although it still allows for management of individual services. Provides built-in middleware for things like Spark, Hadoop, etc. Best usage of Mesos will involve many plugins as it's designed to be easily extended.

If you need fine-grained control over managing your application (insofar as there's a plugin available for what you want to do or you have a team member willing to build one if there isn't) you'll want to use Mesos.

Rancher (and Cattle)

Potentially the best option in that it is itself a superset of the previous two, having an implementation of both. This might also be seen as a downside as more complication in your management software is rarely a good thing and might lead to unforeseen issues.

Rancher features an application catalog that allows for one-click deployment, something Kubernetes doesn't have due to its design philosophy.  However, seeing as Rancher has an implementation of Kubernetes, you can use Rancher if you feel these features are missing from Kubernetes.

Cattle is based off of a stack system, where you group related services together. It also houses Rancher Compose, a feature similar to the Docker service of the same name. This is probably the most interesting part of Cattle, the rest of it being fairly standard (although the Secret management is still in beta). If you have experience with Docker Compose you should read up on it here (I don't, so I'm probably not the best person to write about it).

Resources: "What is Kubernetes?", "Overview of Rancher", "mesos.apache.org: 'What is Mesos?'"


GitLab pipeline stop at the stage of pulling an image from Docker hub

I image you have used the dockerExecutor in gitlab-runner. This means you try to use docker-in-docker.
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
I suggest to configure the docker socket binding which might be less secure but very easy to handle.
To register a runner with docker-in-docker support mount the /var/run/docker.sock:/var/run/docker.sock
e.g.
sudo gitlab-runner register -n \
  --url https://gitlab.com/ \
  --registration-token REGISTRATION_TOKEN \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:19.03.12" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock


Creating a Separate Technical and Business Versioning System

It seems to me that your customers should be dictating the versioning. Both teams seem to be heading towards the same thing but only partially understanding the customer impact. For development, when they significantly re-architect components, there is risk because this could corrupt data or leave core features in a malfunctioning state - this should be reflected by a major version number change.

This doesn't mean much to marketing - everything looks the same, so why would the customer notice this change? No need to reflect this as a major change... unless you are a developer, in which case you risk a huge amount support interactions over those bugs.

Similarly marketing is looking at the user interface with the assumption that with a significant overhaul to the GUI, customers might get lost or hate the new user interface. This could lead to significant support interactions helping customer navigate the new UX, so should be reflected as a major version number change to the customer. There is a lot of risks there... Unless you are developer. Then this is purely cosmetic. Everything is still functioning the same on the back end.

So it seems you need to decide what it is that you feel version numbering is for.

Are version numbers used to signal risk to customers?
What source control system are you using
Does Development even need version numbers at all or are commit IDs sufficient?
Does Development plan features against version numbering for the purposes of setting deadlines?

Depending on the answers, it might be ideal to use a single version numbering system led by marketing - but marketing will need to do a better job of understanding the risks of under-the-hood changes. If this is needed as a designator for setting deadlines for change integrations, then two version numbering systems might be approperite:

Side note: git tags would be great for any version numbering scheme you have mentioned.


Getting data from crashed docker container

So what you can do is you list all of your containers with command:

docker container ls -a

And the second field is an image name, you can run this command to get all the read-only layers with their respective directories to search for the file in:

docker image inspect <image_id> | jq '.[0].GraphDriver.Data.LowerDir' -r | tr ":" "\n"

If the file was something written inside the container replace Lower with Upper.

In case you are on Mac the /var/lib/docker is inside the virtual machine and so you have to first enter that before you can get to those directories described in the command above. You can do it using nsenter like this:

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh


Where does CircleCI-2.0 store the ssh private keys?

I still don't get how your config.yml could launch ansible without a step part. Here is my findings, but I'm unsure it is what you're after, but that was too long for a comment.

According to the documentation here you have to add a add_ssh_keys with the fingerprint of your key as seen in the UI.

- add_ssh_keys:
    fingerprints:
      - "48:a0:87:54:ca:75:32:12:c6:9e:a2:77:a4:7a:08:a4"


If you check the Git plugin documentation, you can see that there is a GIT_BRANCH environment variable that you can use in your pipeline.

GIT_BRANCH - Name of the remote repository (defaults to origin),
             followed by name of the branch currently being used,
             e.g. "origin/master" or "origin/foo"

So, in your pipeline:

stage('Push to Bitbucket') {
    steps {
        withCredentials([usernamePassword(credentialsId: '8e2ca0a6-2dc5-4f89-b76a-03076d6b0843',
                                          passwordVariable: 'GIT_PASSWORD',
                                          usernameVariable: 'GIT_USERNAME')]) {
            sh '''
            # Content omitted
            git checkout ${GIT_BRANCH}
            # Content omitted
            '''
        }
    }
}


Terraform AWS AZ error - manually specify AZ?

If anyone else has this issue, here's how I got around it.

Specify a variable with the availability zones you want to target

variable "zones" {
  default = ["us-east-1b", "us-east-1c", "us-east-1d"]
}

When creating your subnets, use this line

availability_zone = "${var.zones[count.index]}"


Is it true to say that both a "shared server environment " and a "VPS environment" can be either IaaS or PaaS?

Usually, Only a VPS hosting can be either.

While an IaaS is always VPS, what's commonly called "shared hosting plan" is PaaS in which a client shares the same hardware and webserver software with different customers (due to sharing a webserver, it's not a VPS --- Virtual Private Server).

In a PaaS (of the subtype Pass;VPS) only the hardware is shared (so it's indeed a "shared server", but only in that sense).


Instead of hard-coding your branch name in .gitlab-ci.yml use $CI_COMMIT_REF_NAME. There are a bunch of pre-set environment variables you can use to get info about the current job.
The gitlab documentation has a list of them: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html


Compare Strings to get Unique values in Jinja2

Yes, this can be done. The following should do the trick:

"{%- for ip, az in seed.iteritems() %}
  {%- if 'us-east-1a' in az %}
    {%- if firstloop is not defined %}
    {%- set firstloop = 1 %}
{{- ip }}
    {%- else %}
{{- " " + ip}}
    {%- endif %}
  {%- endif %}
{%- endfor%}"

To test it with https://cryptic-cliffs-32040.herokuapp.com/, you can use the following JSON:

{
    "seed": {
      "10.18.13.12": "us-east-1a",
      "10.18.37.93": "us-east-1b",
      "10.18.68.147": "us-east-1a",
      "10.18.21.55": "us-east-1b"
    }
}

You might give the documentation a read if you are going to be doing much jinja templating. I have found it indispensible.


Microsoft DevOps: build triggering vs release

Do not trigger Build 2 automatically from Build 1. Instead add a last step to Release 1 to queue Build 2. There are a few ways to do it, e.g. https://stackoverflow.com/questions/52821204/how-to-queue-build-pipeline-as-task-from-release-pipeline


How to know where a docker image came from?

Typically, if it's from a registry, you see that in the name of the image (if it's not from the default registry, Docker Hub). Images from Quay look like quay.io/<namespace>/<image name>:<tag>.

Some images are from from anywhere else at all. For example images that are FROM: scratch and images created via docker import will not be associated with any registry. These are the famous none:none images (no repository and no tag). In this case, you only have the ID (SHA) and you need do be Sherlock Holmes (or Jessie Frazelle) to match it with something.

So, you can only see what is written in the manifest. Should you trust it? The manifest can be signed, using docker content trust even if the image contents themselves are not.  If you need more certainty on the provenance, you probably need notary  to ensure that trust is maintained throughout the delegation chain.

Hope that answers the question.


The terraform import command uses the "Resource Address" syntax, which is a way to talk about objects in a configuration from outside of that configuration. (This is as opposed to references in the main Terraform language, which are always resolved in the context of a particular module.)

To refer to a resource that is declared in a child module, you can add a module path to the beginning of the address:

terraform import module.assets.aws_s3_bucket.machine-learning BUCKET-NAME


How can I automatically post Codefresh build results to pull requests in Bitbucket?

One could update the build status in bitbucket as follows:

  Adding a build result to a commit

  To associate a build result with a particular commit, you need to POST
  a JSON object to the build status REST resource at:

  https://<bitbucket-base-url>/rest/build-status/1.0/commits/<commit-hash>

  The format of the JSON object that should be used as the request body
  is:

{
    "state": "<INPROGRESS|SUCCESSFUL|FAILED>",
    "key": "<build-key>",
    "name": "<build-name>",
    "url": "<build-url>",
    "description": "<build-description>"
}

One could run such a snippet by codefresh at the end of the build using a script to notify bitbucket regarding the build status of a commit.

  One green build

  The following curl command will add a build to the commit
  9e72f04322c4a1f240e0b3158c67c3c19cdd16e7:

curl -u username:password -H "Content-Type: application/json" -X POST
http://localhost:7990/bitbucket/rest/build-status/1.0/commits/9e72f04322c4a1f240e0b3158c67c3c19cdd16e7
-d @build0.json

  Where build0.json contains:

{
    "state": "SUCCESSFUL",
    "key": "REPO-MASTER",
    "name": "REPO-MASTER-42",
    "url": "https://bamboo.example.com/browse/REPO-MASTER-42",
    "description": "Changes by John Doe"
}


How to organize infrastructure as code projects?

The layout of your repository depends in many ways upon the context you are developing the automation in.  If, for example, you are building out the infrastructure for a product as part of a product team, then it would make sense to tie the infrastructure to the product - i.e. keep the infrastructure in the same repository as the software source code.

If however, you are building out common infrastructure components to stand-up basic infrastructure for DNS, File Storage, Email, etc. then it probably makes more sense to have a single common-infrastructure repository.  Some of this will be tied to the tool you are using, Terraform for example leads you down the path of having a single repository for all of your environments to support the terraform.tfstate construct.

Reusability

You quite rightly called out reusability and code sharing as a problem you are going to encounter sooner rather than later.  This is an important principle in Software Development called Don't Repeat Yourse - or DRY for short.  Most all DevOps tools allow you to modularize your code in such a way that you don't need to copy/paste code:

Terraform Modules
Ansible Roles
Chef Libraries
Jenkins Plugins

Conceptual Consistency

I advise that you don't use too many tools, try and standardise on a single tool rather than changing tools for each project, typically you see sets of tools being used across an organization:

Jenkins, Terraform and Ansible
TeamCity, Octopus Deploy and Desired State Configuration

This makes it easier to share code across projects, document what has been done so far and handover the code to other or future developers.


Let's start from the beginning.

First, I'll answer the original question, the one in the title.

When to put all containers in a pod and when to deploy them individually?

This depends on what your app does. For instance (this is an actual example from the company I work for), for a web application, you may have a pod which runs three containers:

app: the actual app with its code
nginx: serves all the requests
heka: a log service which proxies the logs to wherever you want them

The reason for grouping this three containers within a single pod is simple: a pod assures you the containers run in the same Kubernetes node, this is to have the least possible latency with requests being interchanged along the three containers.

That being said, this is only an example, you could have different and multiple reasons for grouping containers in a pod, but I think this is the most common.

With that being clarified, I follow to the question that you present in the body of your post.

How do you access a single container within a pod?

This is simply using kubectl, I use the command like this:

kubectl exec -it <pod-name> -c=<container-name> -- /bin/bash

Suppose we want to access the nginx container inside our my-app pod; we would type this:

kubectl exec -it my-app -c=nginx -- /bin/bash

If you don't specify the container with the -c flag, it defaults to the first container that is in the Kubernetes yaml (I think). So when you say it was impossible to access a container, you actually did access a container (the default one), because a pod cannot be accessed, it is merely a collection of containers and its configurations.

Hope I've been helpful.


AWS elb vs google cloud for multiple small nodejs mini-apps deployment

you can try google cloud run. here you can run containerized applications on a fully managed serverless platform. You might have to containerize your application.
But it can help you reduce the cost as  the min container can be 0 (zero) and if any to your app is there cloud run can automatically scale to meet the demand.


What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect?

An architect is somebody who thinks and creates designs.

A consultant is somebody who gets hired for things like "Teach us how to do so and so", or to answer questions like "What's your opinion about this design?".

An engineer is the one who is objective oriented and pragmatic, something along the lines like "How can I get that design (created by the architect, and validated by the consultant) implemented and make it work"?.

Their salaries may vary (there are cheap and expensive ones), same for seniority (there are juniors and seniors, while everybody starts as junior, not all of them ever make it to senior). And job demand is pretty variable (by region, over time, economical factors, depending on hypes, etc).

BTW, it doesn't really matter what the actual "subject" is, it could be "IT", it could be "Road Construction", or it could even be "DevOps" ...

PS: What's in a name (or jobtitle), how about a DevOps BA, as shown in this post ...


I don't know of any current plugins for doing this.  That may because a lot of backup functionality is built into MSSQL.

The easiest way to automate a backup via Jenkins may be to use a command line script to use a sql command:

Sqlcmd -E -S touch -Q" BACKUP DATABASE [DBName] TO DISK = 'C:\artifact_directory\my_db_backup.BAK'

You could also also avoid automating this via Jenkins and just create a scheduled backup directly within SQL Server.


Globally configuring Gitlab with Jenkins in docker container environment

yes, it will need access to Gitlab API to report back Jenkins build status.
You can create a service account in LDAP such as Jenkins which can be used to generate Gitlab API tokens and use it to preconfigure Jenkins.
Same as point #2

For more info - https://docs.gitlab.com/ee/integration/jenkins.html


Placing an AWS Lambda in a public subnet

Provided the Public Subnet where you're launching your Lambda is configured with Auto-assign public IPv4 address: Yes it should work in public a subnet without NAT.

Whether or not it's a good idea is another question. In general you shouldn't launch any backend services in the public subnets, the public subnets are for things like API Gateways, NAT Gateways, Load Balancers, etc. Not for launching the actual backend systems. That's the best security practice.

However that aside I'm pretty confident that with auto-assign IPv4 address enabled Lambdas will work even in the public subnet.

Unless you want the Lambda to connect to any VPC-internal resources (e.g. to your RDS) you may just as well launch it without VPC config (leave those VPC and Subnet settings empty). That will give it access to the internet without having to worry about NAT or public IPs.

Hope that helps


Debug Docker networking and Windows 10 firewall

Since Windows use a tiny VM called MobyLinux, the port mapping will happen with tiny VM and not with Windows Host. If you are not reachable from Windows Host to Docker MobyVM, check windows events for possible error. For more networking related information, please see -

https://docs.docker.com/engine/userguide/networking/


How to deploy docker-compose to a remote host

Let's take your questions one-by-one:

  I want to be able to specify different versions of microservices for different prodction servers. How should I approach this?

There are many different ways to handle this.  One possibility be to "tag" your docker images and have different docker-compose files that use these images.

  Should I have one general docker-compose file or one file with specified versions for each production environment?

Depends.  How many environments do you have?  What I've often seen is a single docker-compose that references a separate environment file.

  Where should I keep docker-compose? In a repository?

Again this depends.  A standard enterprise best practice would be to store this in a repository so that it is naturally version controlled, and can easily be accessed by others.

  How it is copied then to a remote host? How to deploy docker-compose.yml file to a remote host?

For deploying a docker-compose.yml it depends on the target architecture.  Since you mentioned Linux you could use scp to copy the file.

  How can I deploy new version of a microservice to a remote host?

The industry leading tool for this right now is Kubernetes.  If you go this approach your docker-compose could be morphed into a Kubernetes deployment.  Using just docker-compose you will need to somehow detect that a new container is available and pull it.  Watchtower or a similar tool may be helpful.

Based on what you've described I would recommend taking a look at the interactive Kubernetes tutorials, and seeing if that is ultimately an easier approach than everything I outlined above.


Can you replicate Active Directory from a Corporate Intranet to an AWS VPC where there is an outbound-only link?

Yes, you just need a Read Only Domaine Controler (search for RODC) in the vpc and you'll be able to have a one way replication as the read only controler won't need to send datas back.

This bring limitation for clients in the vpc, as they won't be able to reach a writable DC they can't change passwords and you'll need a careful site architecture in your forest.


Using cached dependencies for pipelines in Azure Devops

I forgot to add a MAVEN_OPTS variable in mavenOptions:
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'

My pipeline.yml stayed like this:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:

- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | **/pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
      maven
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'clean package'
    codeCoverageToolOption: JaCoCo
    sonarQubeRunAnalysis: false
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'

- script: mvn install -B -e


According to the Cloud Native Landscape, you are looking for a Coordination & Service Discovery:

https://etcd.io/

  A distributed, reliable key-value store for the most critical data of
  a distributed system

https://github.com/Netflix/eureka

  Eureka is a REST (Representational State Transfer) based service that
  is primarily used in the AWS cloud for locating services for the
  purpose of load balancing and failover of middle-tier servers.

https://github.com/alibaba/nacos

  Nacos (official site: http://nacos.io) is an easy-to-use platform
  designed for dynamic service discovery and configuration and service
  management. It helps you to build cloud native applications and
  microservices platform easily.


What is the cleanest branching strategy to use when creating reusable artifacts?

The simplest/cleanest branch strategy is IMHO the one used in continuous deployment: a single/main integration branch which is also your release branch. From What is Your Branching Model?:

  Commits can go all the way to production from one trunk/master, if the
  automated build says the commit was good. It’s the turbo-switch for
  TBD, where no humans can hold up a release by taking their sweet time
  testing it and giving a belated seal of approval. Github, Etsy,
  Netflix (and many more startups) are all here.

You'll effectively use the CI build artifacts from that main branch across all environments. Well, as long they build and pass the respective QA verifications, obviously.

Note: there are no feature branches in this strategy, at least not long-lived ones - in order to eliminate/minimize the integration delays and costs. Practically you can eliminate branch merges from your development.


Is it possible to use a JenkinsFile in one repo to build and deploy a related repo?

Yes, this is pretty easy with Jenkinsfiles with no need for any third-party plugins or anything along those lines: use the built-in Pipeline build step.

I use this to trigger builds of projects in a dependency chain, so that after one project builds successfully, other projects that depend on it will pull in the updated dependency and build against it.

Here is what this looks like in a Jenkinsfile:

build repoName

where repoName is a variable containing the name of the repository you wish to build.

Things can get a little more complicated, such as if you're using folders or Multibranch Pipelines:

build "${projectName}/${repoName}/${branchName}"

This example builds a job within a Multibranch project where the variables are:

projectName - the name of my Multibranch project
repoName - the name of the repository
branchName - the name of the branch to build, usually master

Or this example builds a job with parameters and also triggers the build asynchronously (parent job won't wait for child job to complete before moving on to the next step):

build(
  jobName,
  wait: false,
  parameters: [
    [
      $class: 'BooleanParameterValue',
      name: 'someBooleanParameter',
      value: true,
    ],
    [
      $class: 'StringParameterValue',
      name: 'someStringParameter',
      value: 'some string value',
    ],
  ],
)

As you can see, the invocation of build can get relatively complex.  See the documentation linked above for more complete information.


Google Cloud Registry and multiple projects

Yes, it's possible. Fundamentally GCR is built on top of Cloud Storage, with roughly the same basic access control:

public, i.e completely accessible
private, with quite flexible access control from pretty much anywhere, even from outside the GCP domain - being in a certain GCP project is rather irrelevant.

From Granting users and other projects access to a registry:

  To give specific users or container clusters running on other projects
  permissions to pull images from a registry, you need to grant read
  permission on the underlying Cloud Storage bucket.

  ...

  Select the Permissions tab.
  Click Add members.
  From the menu that appears, fill the Members field with the email addresses of users needing read permission, separated by commas.
  This email address can be one of the following:

  a Google account (for example, someone@example.com)
  a Cloud IAM service account
  the Compute Engine default service account of another project. This account is used by the Google Kubernetes Engine to pull
  container images clusters by default. It is in the form
  [PROJECT_NUMBER]-compute@developer.gserviceaccount.com, where
  [PROJECT-NUMBER] is the GCP project number of the project that
  is running the Google Kubernetes Engine cluster.

There are some differences from Cloud Storage, most notable ones being:

access control applies to the bucket level only, object level access is ignored
GCR can only have a single bucket per GCP project


How do I automate population of known_hosts and authorized_keys?

You have to execute ssh-keyscan. For example to ssh to a host (github.com here ) you have to run below script

# Add ssh key to help cloning private github repo

ssh-keygen -t rsa -N "" -f secrets/ssh/github_rsa
PUB_KEY=$(cat secrets/ssh/github_rsa.pub)
PRV_KEY=$(cat secrets/ssh/github_rsa)

echo "${PRV_KEY}" >> ~/.ssh/github_rsa
chmod 600 ~/.ssh/github_rsa
eval $(ssh-agent)
ssh-add ~/.ssh/github_rsa

ssh-keyscan github.com >> ~/.ssh/known_hosts
echo IdentityFile ~/.ssh/github_rsa >> ~/.ssh/config

echo "Paste the following public key to your host machine ".${PUB_KEY}


Use this source code for reboot your client machine

---
  - name: System Reboot
    hosts: debian
    become_method: sudo
    become_user: root
    become: true
    tasks:
      - name: reboot nodes     # Reboot client side debian machine
        shell: sleep 2 && shutdown -r now "Ansible reboot"
        async: 1
        poll: 0
        ignore_errors: true
      - name: wait for the server to come back  # when server timeout it through error. That error is handle by ignore_errors.
        local_action: wait_for
        args:
          host: "{{ inventory_hostname }}"
          port: 22
          state: started
          delay: 120
          timeout: 200
        ignore_errors: true


What can I use to forward a conection to port 3006 on the bastion to an RDS database?

It is possible on the Linux(Unix) bastion host.

Step-1: First ssh to the bastion host. eg. 16.23.14.20 is the bastion host.

# ssh root@16.23.14.20

Step-2: Create port tunneling by using the below command. eg. The RDS IP is 172.16.13.10

# ssh -L  3006:172.16.13.10:3006 localhost

Verify using netstat command.

netstat -ntulp | grep 3006

Note: If netstat command not available. Using sudo apt install -y net-tools or sudo yum install -y net-tools


Tool or way to support handling dependencies of modules in various platforms (for machine learning in Python)

Python is notoriously painful when it comes to dependency and version management.

I think its your job to make a portable application, and there are usually methods to do so (even if they  boil down to ugly branching logic based on the host OS type/etc).  Most of the time, you can avoid OS specific things with some effort though.

In terms of deploying code to clients, I would recommend looking at tools that bundle your code into a stand-alone application (pyinstaller, py2app, etc).  If you can build a standalone app for your clients for each OS, you won't have to worry about what is on their system (python versions, docker, etc).


Jenkins pipelines vs matrix vs multiple jobs for multiple branches across multiple platforms

You can use the Jenkins pipeline (Jenkinsfile) to run the multiple branches in multiple stages, with the conditional 'IF'.
Below is an example from Jonico. I always use this example to create continuous delivery of monolithic  applications.
https://gist.github.com/jonico/e205b16cf07451b2f475543cf1541e70


Unable to parse YAML: mapping values are not allowed here

You simply have an incorrect yaml syntax in your workflows jobs list.

From what I could very quickly understand on circleCI's documentation, jobs names are accepted as a shorthand if you don't define any parameters. If you have parameters, the list element is a hashmap with top element being the job name. In your specific case, this should give (note the column after the names of jobs with params):

workflows:
  version: 2
  minimal:
    jobs:
      - build
      - tests:
          requires:
            - build
      - release:
          requires:
            - tests
          filters:
            tags:
              only: /^v.*$/


This is something you'd have to build. Kubernetes doesn't natively offer this as a pattern.

There are a few approached you can take, but a common on is to use a queue that has this functionality. I'd suggest starting with Nats because it's a CNCF project that is easy to run natively in Kubernetes. When one of your replicas receives the request, it can post a message to the queue that can deliver the message to every subscriber.


The best practice is to have your web, app and db servers in private subnets and only the actual front end - which is ALB in your case - in the public subnet. The reason is that web, app and db servers shouldn't be accessible from outside even if by accident (e.g. through security group left too open).

Technically it will still work to have both the ALB and the instances in public subnet but it's not recommended for the above reason. Also make sure to lock down access to the instance port 80 and 443 only from the ALB SG if you don't want any direct traffic.

Hope that helps :)


Is it possible to keep all the DSC configurations in Version Control and make Azure Automation fetch it from there automatically?

The article "Continuous Deployment to Azure Automation DSC", or series of articles, walks through setting up a CI/CD pipeline for DSC using VSTS (Visual Studio Team Services).  Building a VSTS account in Azure is free and it provides some number of build minutes for free as well.

Basically, rather than setting up a runbook, you can configure a CI/CD pipeline using build services in VSTS to perform whatever orchestration activities you require for DSC and VSTS enables you to save your code in their source control management system.


Kubernetes - Internal only API Endpoints

I'm not sure I understand why listing the public endpoints in the ingress is messy. The common solution is to list the ingress endpoints in the ingress controller.

As you mentioned, there are alternatives such as Istio, Consul and other network policy solutions. There's also the Open Policy Agent that can help with this, if you don't want to write YAML.


Auto-scaling, how to securely get code from private repo

Option 1:
You could place your private repo in the same VPC as the instance and then pull code from there. Just make sure the repo cannot be accessed from outside the VPC.

Option 2:
Ssh into the new instance and SCP the binary over.

Option 3:
Use docker and deploy a private container registry (e.g. Amazon ECR). Build the image on your machine / build server and then push it to the registry from where it is available to your instances to pull and execute.


Does full automation still provide a better ROI at small scales?

There is definite value in automating manual tasks and placing systems under configuration management that is done using code, not using paper and human intervention.

One huge benefit is the reduction in the amount of rework. You can consider any problem caused by a human mistake in repeating the same sequence a case of rework. As well as any bug that needs to be fixed more than once.

Rework is costly, and there has been some research done to quantify its cost. You can read about it in this whitepaper - https://devops-research.com/roi/

The same whitepaper also quantifies the cost of failure for services/systems, and the slowness to resolve failure. All of which are greatly reduced by having a robust automated process that prevents human mistakes from ending up in production systems.

The topic of automating manual tasks has been also investigated in the State of DevOps Report that has been issued every year since 2013. Most of these are available at https://devops-research.com/research.html as well (with the exception of 2013). In every report it was mentioned as the biggest driver for excellence in high-performing organizations of all sizes. There are plenty of reasons and explanations that can be used as proof for your case.


Increase Kubernetes Nginx Ingress limits for large images in Docker registry

I learned that it's not required to set the ingress class to nginx as I did in kubernetes.io/ingress.class: nginx. All annotations prefixed with nginx.ingress.kubernetes.io apply to the default Kubernetes ingress. Setting the timeouts is also not required. The only important annotation is

nginx.ingress.kubernetes.io/proxy-body-size: "0"

because the default value is way too small for Docker image layers. Without this annotation, we get 413 Request Entity Too Large errors.

The main problem in my case was somewhere else: We had a HAProxy loadbalancer in front of our Kubernetes workers. Being new in those HA technology, it's configuration was pretty default. It contains the following directives:

defaults
    # ...
    retries 3
    timeout connect     1000
    timeout client      1000
    timeout server      1000

According to HAProxys time formats, this got interpreted as ms. In our testing environment, I increased both client/server timeouts to 600s. This fixed all our pushing problems.


Does a slot swap cause an application to restart when using deployment slot settings?

if both are running 'warmed up' and in 'Always On' mode then the swap would not cause a restart of any kind. But go to this URL and read the point #1 and if any of these things are done then it will restart.
https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots#what-happens-during-a-swap


How can I add a when clause for a specific upstream trigger in a Jenkins declarative pipeline?

Is there an 'UpstreamCause' equivalent to the example for 'UserIdCause' that would allow me to specify which upstream cause should trigger a specific stage?

Unfortunately, not in native Declarative syntax.
As you can see from the source code, UserIdCause is special-cased, but no other cause is:
    static boolean shouldRunBeAllowed(Cause causeClass, String cause, String detail){
        if( causeClass instanceof Cause.UserIdCause && Cause.UserIdCause.simpleName == cause){
            return detail == null || causeClass.userId == detail
        }else {
            return causeClass.class.simpleName.matches("(?i)\\.*${cause}.*")
        }
    }

So if you want to trigger conditionally based on which job triggered the build, you'll need to use Scripted syntax. For instance (I haven't tested this so no guarantees it will work without debugging):
script {
  def myJob = 'put-your-job-name-here'
  def causes = currentBuild.rawBuild.getCauses()
  def triggeredByMyJob = causes.any { cause->
    cause.class.toString().contains("UpstreamCause")) && cause.upstreamProject == myJob
  }

  if (triggeredByMyJob) {
    // put steps to execute when triggered by your job here
  } else {
    // put steps to execute when NOT triggered by your job here
  }
}

However, there's potentially a much easier way, although it's not strictly what you asked for:  Give your downstream job a boolean parameter, for the sake of example let's call it runMyStages, and use a when clause to conditionally execute stages based on whether runMyStages is true or false.  Then in your upstream jobs, you can set that parameter in your build step: build(job: 'downstream-job-name', parameters: [booleanParam('runMyStages', false)]).


Stop Multiple Windows Service Simultaneously

The -name parameter of the Stop-Service command is an [array strings][1]. You can pass multiple services to it.

Stop-Service -name service1,service2,service3

Edit

[string[]]$services = @('service1', 'service2')
Stop-Service -Name $services -NoWait
while ((Get-Service -Name $services | Where-Object { $_.status -ne 'Stopped'}).Count -ne 0)
{
    Start-Sleep -Seconds 5
}


How to gain console access to a MySQL server running on a docker container from the host?

Solution:
root $
root $ mysql -h localhost -P 3306 --protocol=tcp -u user -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 17
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | user             |
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)

mysql>

per:
https://stackoverflow.com/q/33001750/4531180


After much tryings, i was look for problem with implementation of solution and reads documentation of resource, and to my surprise that requirements are not attends.Só i   change that configurations mistakes and the storage gateway is flying.

The misconfiguration was about requirements, the minimo indicated by AWS is 16GB of memory, and at least 20% local storage used to cache  all data used loaded to AWS storage.


I can only speak from personal experience rather than best practices, but in our company we'll make a duplicate jenkins instance, then upgrade that. We do main upgrade first, then plugins second, then run testing to make sure our core jobs still work. If all is good, we'll then upgrade the real jenkins during off-hours.

We have 90+ jobs so testing them all is unrealistic - we made an effort to identify 8-10 jobs that span 90% of our use cases and run those on the standup jenkins to affirm everything is healthy first.

I'd highly advise doing something similar if you're using your jenkins in production scenarios. You never know what might break - we had one upgrade where a bug spun up 150+ aws ec2 instances in ~10mins instead of the 3 jenkins actually needed... another time all our unit and e2e testing of our code base broke because it turned out one of the output parser plugins had upgraded to become incompatible with our output format. We've even had our Bitbucket integration go sour and stop kicking off jobs on every push.

Sometimes upgrading all plugins at once can cause confusion - we had one scenario where something broke and finding the cause was difficult because it could have come from one of two different plugins. We ended up spinning up two additional jenkins and upgrading either plugin to identify which was the cause. In general though, upgrading everything at once will pretty quickly show its flaws with a little testing.

No matter what you decide, take a backup first!


Azure ARM Triggers (e.g. Resource Created, Resource Removed, etc)

So you can use something like Azure Event Grid where you can listen to events from different types of Azure Resources and then that event could be listened to by a Webhook trigger which then can trigger your pipeline, see here:
https://docs.microsoft.com/en-us/azure/event-grid/overview


It is not allowed to use stages more than once [Jenkins_Declarative_Pipeline]

Got answer from StackOverflow and here is the link to the answer.

stage('Test') {
    parallel {
        stage("Test_A") {
            stages {
                stage("Tests_A") { steps { echo 'from A' } }
                stage("Archieve") { steps { echo 'from Archieve' } }
            }
        }
        stage("Test_B") {
            ...
        }

        ... and so on
    }
}


Listing filesystem differences between a Docker image and one of its ancestors

There was already a stackoverflow question about that topic, but the answers didn't really help me. The container-diff tools was the most promising solution, but I would have preferred not relying on a third-party tool.

Finally, since docker can export an archive of a container filesystem, I end up with this solution:

BASE_CID=$(sudo docker create ubuntu:18.04)
TARGET_CID=$(sudo docker create rogueimage:latest)

sudo docker export "${BASE_CID}" | tar tv > base.filelist
sudo docker export "${TARGET_CID}" | tar tv > target.filelist

Then using your favorite diff tool you can examine the difference:

vimdiff {base,target}.filelist


How are declarative distros different than Kickstart?

AFAIK Kickstart is only usable for the initial installation of the OS, but not for subsequent package maintenance upgrades/downgrades.

Unless I miss something (quite possible, I don't have much experience w/ them) the Linux distros managed by configuration management tools require a minimum level of health from the OS (and some package requirements, at least in some cases) to be able to operate those tools, say to perform a rollback. At least NixOS appears to not have such requirement (or rather it is already fullfilled at the OS level). From About NixOS:

  Rollbacks

  Because the files of a new configuration don’t overwrite old ones, you
  can (atomically) roll back to a previous configuration. For instance,
  if after a nixos-rebuild switch you discover that you don’t like the
  new configuration, you can just go back:

$ nixos-rebuild switch --rollback

  In fact, all old system configurations automatically show up in the
  Grub boot menu. So if the new configuration crashes or doesn’t
  boot properly, you can just roll back by selecting an older
  configuration in the Grub boot menu. Rollbacks are very fast: it
  doesn’t involve lots of files having to be restored from copies.


jar file is not displaying in docker container

You are using docker wrongly in this case IMHO and I have a serious doubt that your docker exec command is targeting the correct container because, from what I see (and my very limited knowledge of sbt/scala), it should be dead by the time the jar is published. Let me explain.

sbt publishLocal is not a long living command. It will exit when the jar is published to the local .ivy store on disk. So when you launch a container from your image in foreground with docker run --name mytestcontainer myproject:v1 (I added a name for easier debugging purpose on your side...), I am pretty sure you see all your above logs and that the command exits back to your shell.

At this point, if you issue a docker ps, your mytestcontainer will not show up, unless with docker ps -awhere it will be listed in the exited container. If you try to start it again (docker start mytestcontainer), it will exit as soon as the jar is re-compiled/published. So there is no way to target an exec on this container since it is not running. The only way to have a shell in a running container out of this image is to override the start command in docker run but this will bypass your jar publication.

Now, what you first want is to publish the jar as part of the image, not to compile/publish it everytime you start a container out of the image.

FROM hseeberger/scala-sbt:11.0.2_2.12.8_1.2.8
MAINTAINER Sara Waheed <sarawaheed3191@gmail.com>

WORKDIR /myproject
ADD . /myproject

RUN sbt publishLocal

In this case, the jar will be published during the build

docker build -t myproject:v1

You can then run a container out of your image dropping to jshell as intended by the base image you are using:

docker run -it --rm --name testcontainer myproject:v1
May 12, 2019 8:31:33 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
|  Welcome to JShell -- Version 11.0.2
|  For an introduction type: /help intro

jshell>

Or if you want a more classic bash shell, you can override the start command:

docker run -it --rm --name testcontainer myproject:v1 /bin/bash

To go further, if your intent is to deliver your application in a ready to use image, you should have a look at Docker multistage builds. The idea is to build your jar in an image containing all the needed build tools and to copy the produced artifacts to an other image intended only for run that you will finally distribute. Here is the basic idea from your example. Image/command used are solely for illustration purpose and the process is highly perfectible (e.g. harcoded version....):

FROM hseeberger/scala-sbt:11.0.2_2.12.8_1.2.8 as build
WORKDIR /myproject
ADD . /myproject
RUN sbt publishLocal

FROM openjdk:8-alpine
WORKDIR /myproject
COPY --from=build /root/.ivy2/local/com.test/myproject/1.0.1/jars/myproject.jar .
CMD['java', '-jar', 'myproject.jar']


what's the final output image in multistage build?

The final image will have all the layers from diamol/base as well as two additional layers - one for COPY and one for RUN command.
File /build.txt will have the following contents:
Building...
Testing...
Done...


Where are dunder dictionaries stored in salt?

There are currently 5 dunder dictionaries. They are available at runtime and generally not stored statically.

__opts__    - Master or Minion configuration options; stored in configuration files of master and minion, collected at startup

__salt__    - Execution module functions (i.e. __salt__['test.echo']('foo')); from built-in and custom execution modules stored in _modules/

__grains__  - Grains (i.e. __grains__['os']); collected from minions, dynamically generated, or stored in minion config or /etc/salt/grains

__pillar__  - Pillar data (i.e. __pillar__['foo']); collected from the pillar

__context__ - Context defined in state modules and execution modules

You generally don't need to use them, unless you are developing Salt or writing custom modules, runners, returners, or external pillars.


What is the AWS equivalent of Azure Management Groups?

There is no direct equivalent to Management Groups in AWS.
You can achieve some of the same functionality by using AWS Control Tower, specifically around access controls.  I.e. you can set up groups and assign roles to any accounts in an AWS Organization on a per-account or per-group basis and have them deployed to child AWS accounts.
However, there is no such equivalent around, for example, resource allocation.
You can achieve some of the same functionality by using CloudFormation, or by using CT IAM policies policies with specific restrictions, but it's a much more convoluted process than with Azure.


How can i check if all CustomResourceDefinitions have been created properly?

This command works: kubectl get CustomResourceDefinition | awk '{ print $1 }'

You can use -o yaml or -o json


Adam Smith vs. fullstack developers - and productivity in DevOps?

There are two types of work:

Exploitation - Well defined work that can be easily divided into well-defined stages, where each stage can be learned and mastered on its own and handover between stages does not require communication.
Exploration - Undefined work, which requires learning and experimentation to accomplish each stage and handover between stages requires massive amounts of communication of all learning and status of the project.

Adam Smith concerns himself wholly with exploitation and not at all with exploration. The work done in Research & Development departments of the industry is by its definition mostly exploration and so it is not covered in any way by Adam Smith.

But we have seen that in later continuous improvement stages, which are partly exploitative work, the application of CI/CD can bring similar gains in productivity, which could in some way be probably traced down to Adam Smith by someone very imaginative.


Jenkins pipeline fails with No such property error

The issue was with one of the options present in global security configuration “ Enable script security for job DSL scripts”, which needs to be unchecked.


Gitlab CI Runner command line doesn't care about --parameters

You're missing the --non-interactive tag to your registration command.

Full command should be:

gitlab-runner register --non-interactive --locked false --run-untagged true --tag-list java --name foo --registration-token %token% --url https://myurl --executor shell

Relevant documentation for non-interactive registration.


I think you have fundamentally misunderstood the relationship between plans, stages and jobs in bamboo.

A good rule of thumb is to have a single repository of code, which is linked to a single build plan, which produces a single independent module.

You may initially start out creating a single stage with a single job inside each single plan where you just run a single maven command and the unit tests built into your maven build may be perfectly sufficient to exercise your modules completely in this extremely simple use case you probably don't need a tool like bamboo or Jenkins to deal with your builds.

However, very quickly your conceptualisation of a "build" gets very complicated. As you mentioned in your question you might quickly amass multiple modules that are interrelated, or you might need to support your code on multiple platforms or in multiple browsers, and if you get all the way to the CD end of the work flow you might even need to deploy binaries into environments as the result of a code change.

This is where you would want to use the extra orchestration capabilities of your CI tool.

For example, if you are developing a module that you expect to load in a browser then your might have a build pipeline that looks like:

Check in Code -> Build/Compile/Unit Test -> Deploy to a CI environment -> Stand up a half dozen browsers and verify that they all still work correctly.

In the above work flow you would have a single stage to handle your compilation and unit testing which would spit out a compiled or built artifact. Then you have a stage that does whatever needs doing to make your module useful, ie it might create your final shippable product by combining your module with other modules built from other plans, or it might package some software into an RPM, a DEB and so on, depends what you're actually building. If you need to ship to multiple platforms then this is your first stage where parallelism becomes meaningful.

Alternatively, if you are building for a single platform but for multiple consumers you might follow up this build of your component with a bunch of headless browsers kicking off test cases to validate that your changes didn't break anything. These can all happen in parallel on different agents running in different OS's, depends how you have bamboo configured.

Then finally, assuming you are shopping something more complex than a single module, you will want to do a bit of integration testing with all the other components that you need to interact with. This step will usually be separated into a build plan of its own, triggered by the successful completion of the plans which are spitting out individual modules. Here you get a chance to run all the same tests but with your latest version running along side everyone else's module so you can be sure that there are no breakages caused by dependencies.

So at any one time you only have a single repo triggering a single plan. If you have multiple repositories feeding into the same plan and they are completely unrelated but use the same build pipeline then I'd recommend taking advantage of bamboos plan branches so that you can reuse the same build process but with multiple different code bases (if that fits your use case)


If you're using Jenkins pipeline, you can use the email notification post action, as a way to send the message:

post {
    failure {
        mail to: 'team@example.com',
             subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
             body: "Something is wrong with ${env.BUILD_URL}"
    }
}

(you would have to configure the server and credentials for your case)

The message itself can be constructed from the job data. The easiest way is perhaps to tee the output of the job to a file and use it as an artifact to attach to the message that gets sent.


It is convenient to use Environment Variables to provide runtime configuration for such scripts. Especially when it contains secrets that you don't want to hold in any persistent storage or in version control.

When you do actually have secrets, it is even better to use some kind of secret management system that will provide these secrets on request. One such tool that I highly recommend is HashiCorp's Vault.

Depending on how you are executing the jobs on the Bamboo Server, another option might be to store a local file on the server and reference that during builds. This is not really recommended since the server then becomes a special snowflake that only a few people know how to actually set up. It is much better to allow malfunctions with the server, or even the loss of all server data, to not affect your ability to deploy code. Thus having special files on the server itself is not so great.


How does Kubernetes terminate an instance after helm upgrade

Suppposing that you are using typical Deployments, a new replicasets is created and the old one shrunk.
Looking at the docs for 1.16, it seems that TERM is sent, and then finally KILL.

When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container. Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server. If the Kubelet or the container manager is restarted while waiting for processes to terminate, the termination will be retried with the full grace period.

https://v1-16.docs.kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
Therefore, you'll want to accept TERM signals, not just KILL. Also, you'll need to ensure the signal is properly transmitted from the surrounding docker container into your server process.


Is it possible to force to false or true the App.config transformation with arguments on Azure DevOps Pipeline?

Short answer, yes.

Azure Pipelines supports conditional steps. Find these under Control Options in any build step.

How you build your custom conditions will depend on what triggers your step to true/false.

The screenshot has an example of how we control a build step based on a build variable (defined in the variables tab of the pipeline).


How to get EC2 instance tags when pulling CloudWatch data?

CloudWatch metrics contain "dimensions" that loosely match InfluxDB's concept of "tags". The list of dimensions available for EC2 metrics is available in the AWS documentation. Instances tags are not available and I'm not sure what you mean by env_variables but since all that is supported is

AutoScalingGroupName
ImageId
InstanceId
InstanceType

the answer is still going to be this is not possible.


How to create the smallest working docker image every time?

A Dockerfile creates a new layer for each of the commands in the file. Since layers are well, layered on top of each other - you cannot remove files that a previous layer added. This is why when you install packages, or download files, or create builds each in a separate command - these are still there in the image, even if in a future layer you removed them.

So if you just change this:

RUN apt-get update -y
RUN apt-get install -y wget a-package
# ...
RUN apt-get purge -y wget
RUN rm -r a-build-dir
RUN apt-get purge -y a-package

To this:

RUN apt-get update -y \
    && apt-get install -y wget a-package \
    && mkdir a-build-dir \
    && wget http://some-site/very-big-source-code.tar.gz \
    && tar xzvf very-big-source-code.tar.gz \
    && do-some-compilation \
    && apt-get purge -y wget \
    && cd .. \
    && rm -rf a-build-dir \
    && apt-get purge -y a-package

You will get a much smaller image.

Another option, is to squash the image after you built it.
Q: How does the new docker --squash work?

Yet another option, is to choose a slim base image. For example, images that use Alpine Linux as their base instead of Debian, take just 10-15mb instead of 180-250mb. And this is before adding your own application and data. Many official base images on Docker Hub have an alpine version.


You may use matchbox on a VM as you do on a baremetal machine, you won't be able to use packer on a baremetal machine ont he other hand as it doesn't handle any PXE boot option.

That said, leveraging vSphere/AWS/ API/cli to create a new machine from a template is usually quicker and more effective than using the API/cli to create the VM and then make it boot in PXE.

Packer will create machines, matchbox won't, it will allow you to match a profile with a machine but it won't rack your machine nor plug the networks and power cables, it won't do the machine creation neither for a VM, that not its goal.

So the main difference is the "before provisioning (chef)" step, as the machine creation, Packer will handle the machine creation whereas matchbox won't, matchbox will handle booting a blank machine and push the OS where packer usually won't.


How to connect GCP Load balancer to Kubernetes cluster(single node cluster)

You can use the GCP HTTP(S) Load balancer. Since you are running a kubernetes cluster (Assuming you are using GKE), GKE uses Google compute VM instance groups which can be put behind a load balancer as target group.

You can follow the below steps to setup a load balancer and put your kubernetes node behind a load balancer.

Login to GCP console and go to NETWORKING > Network Services > Load balancing
Click on Create Load Balancer at the top of the page.
Click on Start Configuration for HTTP(s) Load balancing
Select From Internet to my VMs if you want to access it from public internet and click continue
Give a Name to your load balancer
Click on Backend Configuration and Select Backend services (from drop down) > Create a backend service
Give a name to backend service and select the other parameters appropriately. Select Instance groups in Backend type.
In the New backend section, select the kubernetes instance group name from the drop down. After selecting the proper instance group, you can enter the ports which will accept the requests. You can modify other things if required. At the end, click Create.
Next you can setup Host and Path rules if you need any.
Next, you have to configure Frontend configuration which will be your frontend to accept the requests on the port and send it to the backend target group which we configured in the step 8.
Once all the configurations are done, click on Create to create your load balancer.

If everything was setup correctly, you should be able to access the contents from the load balancer endpoint.


The critical question is - can you teach your job #2 to patiently wait for job #1 to complete when it gets to that dependency point and appropriately proceed or bail depending on job #1 result?

If you can't, then the answer is no - you'll have more trouble dealing with #2 failures due to incomplete job #1, too much noise in the process.

If you can (and if you have enough resources) - yes, by all means - your process will be faster. You might not even need special support from the CI system for such dependencies, it could be completely oblivious, it would simply see the 2 jobs as parallel.

A middle-ground possibility, if your CI systems supports it, would be to stagger the jobs: start job #2 with a delay sufficient to bring the chances of job #1 completing before job #2 needs it in a much more comfortable range (say 99%). The delay would be plain eyeballed or empirically determined, either fixed or, in a more advanced approach, automatically derived from historic measurements/stats (if available).


How to navigate to an index.html with nginx?

Did you only try to reach localhost? Or even 0.0.0.0 or 127.0.0.1?
Depending on the operating system, docker config and the internal networking setup, some problem can occurs to reach the docker network.
You can inspect the container to get the network IP, maybe you can go there https://stackoverflow.com/a/20686101/2904965


How does the creation of an AMI on AWS work and why is the creation time unpredictable?

When creating an AMI, you are making a copy of your EBS volumes. This utilizes block storage and will only need to process the diffs of your EBS snapshots. If you changed a lot of data in your EBS or don't have a previous snapshot, it will take longer to create your AMI.

Here is a quote from the documentation at https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Creating_EBSbacked_WinAMI.html

  Depending on the size of the volumes, it can take several minutes for
  the AMI-creation process to complete (sometimes up to 24 hours). You
  may find it more efficient to create snapshots of your volumes prior
  to creating your AMI. This way, only small, incremental snapshots need
  to be created when the AMI is created, and the process completes more
  quickly (the total time for snapshot creation remains the same).


Which cloud region could have the fastest time for a docker image push?

TLDR

From cloud instance, just go push to docker hub; you will be routed to the best location automatically.

Full details

DockerHub is a massive scale and very critical global service.  So, you almost definitely don't have to think about this at all.

There are techniques on cloud hosts (AWS, Azure, GCP) to replicate & host content/services in regions across the globe and route traffic to the closest entry point automatically.  The cloud hosts make this quite easy to do with their managed services (while without them it would be very difficult).

Also, there are many third party providers of technologies like CDNs which do similar things at scale (and they are really good at it).  They are extremely frequently used in these cases.

So, again - just go to docker hub and don't think about it.  You will be routed to the best location automatically.


One could use the same docker-compose file for dev, test, acceptance and production (DTAP) systems. Things that deviate per environment, e.g. passwords could be defined in an .env file.

If .env files are not a solution for some reason then one could create different docker-compose files, e.g. docker-compose.override (dev), docker-compose.test, docker-compose.acc and docker-compose.prd.

I personally prefer the first option, e.g. using .env files and a docker-compose.override for dev.


Different execution contexts for same POM - use properties files?

I'm not an expert on this, but maintaining separate pom files seems like a recipe for things to get out of sync.

It would make more sense to define specific dependencies within the same pom.xml and execute them depending on specific goals or targets.  I.e. -Pdev.

You could also then define a list of shared dependencies used by all targets (i.e. chrome driver or what not), and specific dependencies for each target.


Ansible tmp files created with odd permissions

The reason for the failure is that the server in question, unlike all the others, had sftp disabled.
Why the error messages are what they are, I don't know (why the errant scp transfer mechanism failed?), but I don't have time right now to investigate with the debug option enabled.

But that was the issue, and adding scp_if_ssh=True to the [ssh_connection] section of my ansible.cfg solved the issue:

[ssh_connection]
scp_if_ssh=True


Interacting with a Trident container (NetApp) via CLI within a Openshift/Kubernetes cluster?

There's apparently 2 methods one can use to leverage tridentctl to interact with a running Trident Pod in their Openshift/Kubernetes cluster.

1. Server string

The tridentctl CLI can be instructed to talk to whatever server you want remotely using the -s or --server argument. In this context you could use the approach of remote shelling into the Trident Pod and then directing it to interact with the local Trident Orchestrator daemon.

For example

$ oc -n trident rsh trident-6bdbdbc5dd-l8qtr
Defaulting container name to trident-main.
Use 'oc describe pod/trident-6bdbdbc5dd-l8qtr -n trident' to see all of the containers in this pod.
/ #

Now to interact:

/ # tridentctl -s 127.0.0.1:8000 get backend
+--------------------------+----------------+--------+---------+
|           NAME           | STORAGE DRIVER | ONLINE | VOLUMES |
+--------------------------+----------------+--------+---------+
| ontapnas_192.168.101.101 | ontap-nas      | true   |     389 |
+--------------------------+----------------+--------+---------+
/ #

2. Creating a Bash Function

To expand on the above method, one could construct a shell function or alias in Bash to streamline this interaction like so:

Openshift

tridentctl () {
    oc rsh -t -n trident -c trident-main \
        $(oc get pods -n trident -l app=trident.netapp.io -o jsonpath="{.items[].metadata.name}") \
        tridentctl -s 127.0.0.1:8000 $*
}

Kubernetes

tridentctl () {
    kubectl exec -t -n trident \
        $(kubectl get pods -n trident -l app=trident.netapp.io -o jsonpath="{.items[].metadata.name}") \
        -c trident-main -- tridentctl -s 127.0.0.1:8000 $*
}

With one of these functions set, you could then run the following command from your macOS system:

$ tridentctl get storageclass -o json
{
  "items": [
    {
      "Config": {
        "version": "1",
        "name": "basic",
        "attributes": {
          "backendType": "ontap-nas"
        },
        "storagePools": null,
        "additionalStoragePools": null
      },
      "storage": {
        "ontapnas_192.168.101.101": [
          "NA_01_aggr1",
          "NA_01_aggr2"
        ]
      }
    }
  ]
}

How it works

The above function merely determines the name of the Pod within the trident namespace and then remotely executes the command tridentctl using the same method above (from method #1) of connecting to the locally running Trident Orchestrator daemon within the Pod using the --server string.

The advantage with one of these approaches is that you simply can use a function/alias to codify it.

Caveats

Keep in mind that with the above shell function/alias approach will not work with commands such as this:

$ tridentctl create backend -f <backend-file>

because the files you'll be referencing with it, -f <backend-file>, will be local to the system where you'll be running the function/alias, but the tridentctl command will actually be executing within a Pod on another system. This Pod will not have access to your local system's filesystem.

Your only options in this situation will be to either copy the file into the Pod's /tmp filesystem and execute it from there or to redirect the contents of the file, <backend-file> through a pipe, (|), for example:

$ cat <backend-file> | tridentctl create backend -f -


Cannot access service using docker-swarm, but without

The problem was that the vpn network has the subnet 10.0.0.0/8 and the ingress network of docker swarm has the subnet 10.0.0.0/24.
So there were a conflict.
Fix:
Changing either the vpn subnet or the ingress network subnet.


What is the difference between feature flags and feature toggles (if any)?

If you take it on an objective only point of view, there's no difference.

But feature toggles carry a heavy binary "on/off" for the whole application idea, whereas feature flags are a more relaxed term which can encompass ramp up testing more easily.

When you launch a ramp up testing and turn the new feature on for 10% of your traffic or user base and move up to 20% the next week and so on, the toggle term become counter productive.
Some people (managers/commercials/some client support people) may not really understand only a part of the clients get the new feature.

Ramp up can be defined as (quote from wikipedia on software testing technology):

  The usual sequence is to ramp up the load: to start with a few virtual
  users and increase the number over time to a predetermined maximum.
  The test result shows how the performance varies with the load, given
  as number of users vs. response time.

I like etsy's feature API (even is not maintained) as source for explanation, it's an easy to undertand framework which has a good readme around feature flags and their uses.

So the difference can be done in the usage of the terms to just help avoiding confusion on what happens on the application.


Multiple apps (microservices) and one proxy (nginx) docker-compose configuration/architecture

This is going to be an opinionated answer as both options are clearly possible.

I would go with the first option you described for the following reason:

It seems that at the moment you don't have a lot of microservices (otherwise you wouldn't be using docker-compose and would likely be doing k8s or at least docker swarm). Now, if your application grows a lot - you would likely outgrow compose in any case, so it would be wasteful if you optimize for compose at this stage anticipating future complexity.

So my take would be to go with option 1 for now which is much easier to implement. And when you need extra flexibility, likely you'd have other reasons to switch from docker-compose to something else.


How to increase the size of a disk on a running instance on Amazon EC2

Amazon AWS just released (on 13th February 2017) a new feature that allows to change a size of an EBS volume.

source: https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/

This allows to increase the size of an EBS volume on an existing instance, while it is running.

It is important to note that changing the volume size, does not change the size of the filesystem on the volume (for most filesystems). Additional steps might be required (depending on filesystem) in the operating system itself. For example a resizefs on ext4 filesystem used by most Linux OS today.

Full documentation from AWS describing the process can be found at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html

In some cases the block device has mapped partitions, and only then one (or more) of the partitions include a filesystem. In this case the partition will need to be resized first, and only then the filesystem. This process is described in the documentation as well.

The new "online" resize feature described in the blog only applies to current generation instances, and there are some other considerations and limitations that need to be checked before attempting a volume resize.