What is the Maven versioning strategy for Continuous Delivery or Deployment?

we now want to release faster

And from your comment:

  Basically, since the artifacts depend on each other, we need to release them in the correct sequence. And since there's a lot of them, it takes some time.

I think you're having an X/Y problem. Yes, versioning plays a role, but it seems like you would benefit from solving the source of your problem first, which is that you have to release something in the correct sequence; presumably you also have to release everything at the same time, with a large downtime of the whole system.

Try to reach a state where you can release new versions of each artifact in arbitrary order. This basically means:

Employing some form of Dependency Injection (which is a big word, but can be something pretty simple, depending on how your software works).
You need to be able to have multiple versions of each artifact on the server at the same time, and to switch between them without much effort. That "switch" should be be on the side of the "user" of an artifact, not on the artifact itself, if you can do so at all.
Wherever you can, it also is great to be able to run multiple instances simultaneously (obviously this is restricted for data management stuff, databases and such).

I'm not telling you how to do that or that it is easy - I don't know your architecture or technical constraints. But this should give you a general roadmap.

When you have that capability in place (at least conceptionally), then you will be able to release smaller parts more often, and a little more independently. And - to answer your actual question - sticking with SemVer is then not the worst you can do.


The stock Azure Pipelines app for Slack does not support that workflow of uploading artifacts to Slack. You could use this marketplace task and upload your release notes artifact with it.

Bonus points suggestion, if you are going down the release notes path I would take a look at using Richard Fennell's Generate Release Notes task in combination with the Wiki Updater task.

You can create a generalized Task Group or yaml template that you could pretty much re-use anywhere. For example in the Wiki updater task you can create a release name wiki page formatted like:
$(Build.Repository.Name)\$(Release.EnvironmentName)\$(Release.ReleaseName).md

And then in your Slack message send a generalized link to your release notes.
Release Notes Wiki link <https://collection-name.visualstudio.com/projectname/_wiki/wikis/Release-Notes?wikiVersion=GBmaster&pagePath=$(Build.Repository.Name)\$(Release.EnvironmentName)\$(Release.ReleaseName)|$(Release.EnvironmentName)\$(Release.ReleaseName)>

You could push them back into the Repo that the release originated from or a central release notes repository. Your release notes get source controlled and are nicely organized!


There are tools around the web for saving desktops. Still best solution I found out to be running a set of virtual machines on your computer, having one setup in one vm.

That takes some space, but you can save the state of vm upon vm shut down and come back to exactly same state when you come back. Maybe sessions have died but the windows certainly are as you left them.

Solution is stated on http://www.makeuseof.com/tag/save-restore-windows-session/ and it is for windows but I suppose the same will apply for Linux vm too.


Multiple Ingress controllers in Kubernetes

I assume you've created your nginx ingress controller in it's own namespace? By default it's called ingress-nginx.

You can create your second Nginx Ingress controller in a seperate namespace so that it's logically isolated from your first. You'll need to give it a different name, but I see no reason why you can't do that.

Doing so will spin up a seperate AWS ELB, which you can then map to your new DNS record and then you can slowly migrate your services over.


How to trigger jenkins pipeline only on merge to develop branch from github webhook trigger?

You'll want to use the CHANGE_TARGET environment variable.  If that variable exists then you are dealing with a merge request.

You can use the following code snipped to then determine the branch name and alter the job's logic depending on which branch the code is being merged into.

branchName = env.CHANGE_TARGET ? env.CHANGE_TARGET : env.BRANCH_NAME

In your code example you would replace

        expression {
         return env.BRANCH_NAME != 'feature_cicd'
         echo env.BRANCH_NAME
        }

with

        expression {
         branchName = env.CHANGE_TARGET ? env.CHANGE_TARGET : env.BRANCH_NAME
         return branchName != 'feature_cicd'
         echo branchName
        }


Strange behaviour of Docker CE/EE client on Windows - timeout for certain image layers

You may wish to try disabling and uninstalling any antivirus software you have installed and see if it resolves the issue (I of course don't recommend this in the long term). I have seen two different instances of antivirus interrupting SSL conections on Windows 10 recently. Chrome would report a similar error to "bad record MAC" and the issue was generalized to any SSL website or connection and any browser. Unstalling or changing A/V software resolved the issue.


AWS Task definition memory for Ubuntu docker image

There are two important parameters that you can use wisely to deal with AWS ECS task definition.

memory:

  The hard limit (in MiB) of memory to present to the container. If your
  container attempts to exceed the memory specified here, the container
  is killed.

memoryReservation:

  Docker attempts to keep the container memory to this soft limit;
  however, your container can consume more memory when it needs to.

I recommend you to go with memoryReservation. So your task definition will look like this.

{
  "family": "my-alpine",
  "networkMode": "bridge",
  "containerDefinitions": [
    {
      "image": "alpine:latest",
      "name": "my-alpine",
      "memoryReservation": 100,
      ...
      ...
    }
  ]
}


You notice that you now have access to 6 availability zones in us-east-1, where you previously only had 4 or 5.

Adding capacity is the reason that new availability zone is now available.

Because of the way the spot market works, there is no way to know what the highest winning bid was, at any given time... because the current market price is based on the lowest winning big for each round.  That sounds counter-intuitive, but here's the logic: given n available instances in the spot market, the system identifies the highest n bids.  These bids are the winning bids.  The lowest of these winning bids is the price all of the winners will pay, and becomes the new current market price.

Each time the market shifts (new bids arrive, or active apot requests are canceled, or on-demand instances are started/stopped, presumably resulting in the available spot pool growing or shrinking), the system re-evaluates all the bids.  If you are no longer a winning bidder, your instance will be terminated, but if you're a new winner, you'll get the instance you've been waiting for.

Based on what's publicly known, we can conclude these things:

the current market price is the lowest bid among all the current winning bidders, assuming there are any winning bidders
you will not get a spot instance right now if you bid below the current market price, but you'll get one if/when the price drops below your current bid.

Therefore, when the spot price is at maximum, there are no unsold spot instances.  We can logically conclude that this is necessarily true, because:

a single winning bid that is lower than the maximum would cause the market price to be lower than thr maximum; therefore, it defies logic that there could be any spare capacity, because this implies that all current winning spot instance bidders made outrageous bids when they could have obtained the instances for substantially less money either by placing a smaller spot bid or by using an on-demand instance
it also defies logic that AWS would artificially inflate the price above the on-demand price, because there is no rational reason for someone to pay more in the spot market when the could use an on-demand instance.

We can further conclude that one of two things has happened to cause this, both of which have the same net result:

there is no spot capacity available at all -- if this is true, it makes sense that the indicated price would stick to the upper maximum value, because you must bid at or above the current price to get an instance now.  The actual market price in a situation where there is no capacity at all would be more properly called NULL because there is no price at which you can obtain a spot instance.  Or, you might say the current market price is infinite, because regardless of how high you bid, you can't have a machine that doesn't exist.  Some might naïvely say the current price is $0 but that is definitely incorrect and would send the wrong market signal.  Where this leaves us is that the most logical representation for the price in a zero-capacity condition would be the maximum/ceiling price.  This gives a clear market signal to any sensible algorithm -- don't bid in this market, the value is not there; or
there is some capacity, but it's all claimed by people who bid the maximum possible.

Which of these two conditions is actually true doesn't matter, because the net impact is the same -- no available capacity.

If that second possibility doesn't make sense to you, that's because you are a rational person.  There are always people who try to game the system.

Instances in the spot market are ephemeral, since the market shifts are outside your control.  I've had spot instances run uninterrupted for many months on end, and I've had them cycle up and down many times over the course of one day.  This is part of the market.

It is possible to improve the chance of a spot instance persisting by placing a high bid... but it is not possible to guarantee that a spot instance will keep running, regardless of how high you bid... because even if all the high bidders colluded with each other and bid the exact same high price, the capacity is finite, so if all current spot instance capacity is sold, not all of the high bidders would have machines allocated to them.

However, there's always that guy who thinks he's smarter than the rest of us, bidding far too high, presumably on the assumption that his instances are likely to continue running, and that since price spikes are transient, he expects only to pay the high price for only a short time.  If there are such gamblers in the market, then they would have the same effect of pushing the price to its limit when capacity is constrained.  This becomes clearest when you consider the fact that if there were only 1 available machine, then only 1 person needs to place an outrageous bid, to set the market price unreasonably high.

The maximum spot price bid is now 10x the on-demand price.  This has not always been the case.  The logical conclusion is that there are indeed those who bid outrageously high, because they are trying to game the system and use spot instances for a persistent workload, so they bid high to try to keep those machines running.  It's a flawed strategy, but it does explain the prices hitting the ceiling and sticking there when capacity is limited.


What is the recommended way to monitor differences between multiple Kubernetes namespaces/clusters?

Edit: The deploy script that is described in this answer is very much  looks like Helmfile  and we are now in the process of upgrading to Helmfile so that we can retire our scripts that run multiple Helm releases as that’s that Helmfile does well.

This is a bit of an alternative and indirect  answer. Use infrastructure-as-code best practices to avoid environments getting out of sync. I believe people will find it a helpful alternative suggestion so a valid answer.

Prevention is better than cure. Don't let the namespaces get out of sync by treating your configuration as code and manage it via git. Helm is a package manager for Kubernetes configuration. It can handle arbitrary configuration objects such as Openshift extensions. This let's you setup a deployment pipeline and a git workflow that ensures your environments stay in sync. (Update: use Helmfile to manage multiple Helm releases).

By way of an example in our setup we have a Helm chart that just installs configmaps and another that just installs secrets. The actual values to install are passed in as a parameter like my-app-secret-values.yaml. So we have different yaml values all under git and a deploy script that iterates over them applying the helm chart that loads the values and updates the corresponding configmap or secret. Helm reads, compares, and only writes when something has changed. So the deploy script will result in only changing a secret or config map that has changed in git. All the configuration is under git and managed like code. Syncing configuration between environments can then be managed just like promotion of code. As a bonus helm creates a backup every version of every object it changes stored inside of Kubernetes. If let's you list the prior versions and rollback to a previous configuration. Also as everything is deployed from git all the history of all settings in all namespaces can be seen in git.

We also install and run out main applications as traditional Helm charts that are promoted through environments. Those helm charts reference the configmaps and secrets that are separately deployed when we want to adjust the settings in a given environment.


The Jenkins website has a blog post on this issue:

But one issue, which we feel is the most significant issue of all, one for which we now believe no resolution is possible: the rights to the name Hudson.
Oracle has told us that they have trademark applications filed in both the EU and US for Hudson, based on Hudson's creation by Kohsuke while working at Sun. The problem is that this trademark ownership gives Oracle the ability to revoke the Hudson project's right to call itself Hudson at any time, and while Oracle has made an attempt to offer some guarantees (most notably, that binary releases of Hudson, once they've been released with the name Hudson attached, will always retain the right to the name), they are not offering any binding guarantee that the Hudson project will be able to retain its use of the name in perpetuity.
Therefore, to continue using the name Hudson means ceding some of the project's independence to Oracle - if the project and its governance board opted to go in a direction Oracle disapproved of, Oracle would be able to take away the naming rights. Or, in a less dramatic scenario, Oracle could insist on certain changes to the code, infrastructure decisions, process, etc, regardless of opposition from the Hudson development community, in order to retain the rights to the name.

The Jenkins team were concerned that using the name 'Hudson' would give Oracle far too much control over the project, because there was a constant threat that if the project didn't do what Oracle wanted, they'd simply yank the rights to the trademark.
In order to retain independence, the developers of Hudson voted to rename the project Jenkins. However, here's where it gets confusing. Oracle chose to continue developing Hudson, essentially making Jenkins a fork because the code started to diverge. There is some dispute about whether this is really a fork or a rename, although it's more an argument over semantics than constructive discussion.
As of now:

Jenkins has become the predominant version, and interest in Hudson has collapsed

Hudson is now managed by the Eclipse Foundation, after Oracle gave up development and donated the code to Eclipse.


Ansible warning that host key has changed, even though I've taken multiple steps to prevent this warning

You can add this in inventory:

ansible_ssh_common_args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

It's ignoring check host and setting /dev/null as Known host file.


How to retrigger a Jenkins build after failure using declarative syntax?

The other answer is incorrect.  There is indeed a builtin to retry arbitrary sections of your job called retry.

If you want to retry the whole job three times, you can wrap your entire job in a retry block:

retry(count: 3) {
  // your job definition here
}

However, if it's safe to do so, I would recommend wrapping individual steps or stages instead:

stage('my first stage') {
  retry(count: 3) {
    sh('some-command')
    sh('some-other-command')
  }
}


after automatic update jenkins fails with `InvalidBuildsDir: .../${ITEM_FULL_NAME}/builds does not resolve to an absolute path`

I'm not very familiar with Jenkins on Windows, but I imagine your issue lies in the fact installing Jenkins on drives other than C:\ are more difficult to deal with. Since Jenkins is not running, you'll have to configure config.xml by hand.
Here's a post on stackoverflow.com that may help you. As they state, you'll probably need to find the Workspace Root Directory to D:\Jenkins\${ITEM_FULL_NAME}\workspace and Build Record Root Directory to D:\Jenkins\${ITEM_FULL_NAME}\builds (or whatever values are specific to your environment) in your config.xml.
The post also mentions the bubble help info that is useful:

This value may include the following variables:

${JENKINS_HOME} — Absolute path of the Jenkins home directory

${ITEM_ROOTDIR} — Absolute path of the directory where Jenkins stores the configuration and related metadata for a given job

${ITEM_FULL_NAME} — The full name of a given job, which may be slash-separated, e.g. foo/bar for the job bar in folder foo

The value should normally include ${ITEM_ROOTDIR} or ${ITEM_FULL_NAME}, otherwise different jobs will end up sharing the same workspace.


You need to

Expose the server port in the Dockerfile
get the IP that the container is listening on (e.g. docker inspect code-rade_cache_1 |jq .[0].NetworkSettings.Networks[].IPAddress ) -- note that you will need JQ installed  on your computer to do this
Set the connection in the mysqlphpadmin


How do I delete a default AKS node pool when I get an error saying it's the "backend address pool of the basic load balancer"

If my assumption is correct that's related to having a loadbalancer that wasn't correctly deleted.
Which could be resulted by redeploying a service (of type: load balancer) in a short time before Azure cleans up the previously associated loadbalancer resource.
To my knowledge, that could be fixed with the rolling of Kubernetes 1.17 and the Loadbalancer Finalizer feature, through this pull.
please update the question if that showed up with the new cluster version.


I think you would be hard pressed to get a straight definite definition of a Platform Engineer.

Where I'm currently positioned everyone on the team starts out with their own special role but very quickly the end-game seems to be to feel comfortable enough to be called, or to call ones self, a platform engineer.

When I was first told that officially a platform engineer the brief explanation of what that meant, for me at least, was that I was going to need to do a whole lot more defending of my platform.

I wasn't quite so focused on helping our developers directly (although that's still the goal obviously) but as the platform owners our jobs was mostly to make sure that our developers where being good citizens by making sure we designed the platform so that we could show them what they did wrong if something happened and we could give them all the information they needed to fix their code.

The obvious implication of this is that our platform is perfect, which it obviously isn't. But at least we can support the developers in a way that gets them working on a solution on their end while we're fixing up our side of the agreement.


py2neo AddressError When Connecting to neo4j in Docker

I had a similar error while using a virtual environment using PyCharm.  In my case, I was using an upgraded version of neo4j-driver in my virtual environment compared to my host environment.  I downgraded neo4j-driver from 1.7 to 1.6.2.

Make sure you have neo4j-driver installed in general.


In my opinion, the DevOps culture come along with a methodology change toward Agile process management.
ITIL is heavily aimed at a clear formalism of the process and the results and thus more adapted to a Waterfall model.

This doesn't mean ITIL is incompatible with Devops, but usually this will be two separate process with different timelines.
I mean that the inclusion of a new product within ITIL referential will usually be delayed until the product/application has been released in production for a while, where early pitfalls and some documentation needed to integrate ITIL has been done and adapted after the product is "live".

One of the thing in ITIL is the Service Design, which is assumed to be defined before any development task, an agile process will/may review the design in each iteration, breaking the formalism needed in an ITIL process.

The main goal of ITIL is, as you said, to provide a framework to ensure nothing is omitted between the design/conception and maintenance phase (Build/Run).  In a devops culture, the whole team is responsible of all phases on long term, hence why the formalism is reducted.

That doesn't mean we have to forget ITIL, the core principles are absolutely good and, in my opinion, should be used as a checklist to build the initial backlog of a product.
It's just that following the ITIL principle with all its formalism goes against the time to market reduction goal of a quick iterative software development and sometimes is not even applicable as there's less transmission of information needed between teams, as the tasks are done by the same team.


Docker Container always stopped and started

You should keep at least one service running in foreground to keep the container running, you can do that using ENTRYPOINT or CMD or both. For more info check Dockerfile: ENTRYPOINT vs CMD.

You can also use a docker image for pm2 from docker hub, that will save your time and you can check the Dockerfile for it to see how it is written.
Check the keymetrics/pm2-docker-alpine project, for example.


AFAIK neither OpenShift nor Kubernetes itself allow you to externalize services configuration. You should have a bunch of yml/json files somewhere and control their deployment/configuration using CLI tools.

But there is Rancher which in version 2.0 allow us to import external Kubernetes clusters and manage them as Rancher's own environments.

Description of how this could be done with Rancher you can find here.

FYI: Rancher 2.0 is slowly moving to Beta so it might take time for you to use it in your activities.


How to prevent that tags will be overwritten in docker private registries?

It is possible to prevent the overwrite of tags by listing the existing tags using https://github.com/jessfraz/reg and exit the pipeline if there is a duplicate.


Does the following work?

docker-compose exec mariadb /usr/bin/mysql -u <your_db_username> --password=<your_db_password> < your_db_backup.sql

Note: mariadb is the name of your container.

Edit: I realize I'm using docker-compose. So I'm not entirely sure if this would work with just a Dockerfile.


Where can I find professional insights into the impact of DevOps upon SecOps?

As alluded to by Tensibai ♦, I think the answers you are looking for start from my answer to "What is SecOps?", specifically from the "Further Reading" section:

DevOpsSec by Jim Bird
DevSecCon: A DevSecOps Conference
Velocity EU: Continuous Security

Of the above I am currently reading Jim Bird's book, have had the pleasure of working with the organiser of DevSecCon in a previous role and had the privilege of attending the Velocity Session in Amsterdam last year.

I would advise you to reach out to Francois Raynaud and ask the question of him, he has worked in the space for a long time and is really helping the community evolve as part of DevOps.

Beyond my links, the O'Reilly Security Conference has published videos on Safari Books Online:

2016

New York, NY
Amsterdam, Netherlands
San Francisco, California


Ansible - when statement task always return false value

Your statements look ok. If I create a test case from the sparse info you provided and what I can observe when gathering facts from my own hosts, I can detect old/recent distributions correctly using your conditions. See my example below.

I suggest you double check the data gathered from your host to understand why it doesn't match. The easiest way for that: from the same host your use to launch your playbook, run

ansible -i path/to/your_inventory.ini your_target_group_name -m setup -a "filter=ansible_distribution*"

Here is the test.yml playbook I wrote to test your (known-so-far) values and your conditions

---
- name: Statements for checking old/recent distribs
  hosts: localhost
  gather_facts: false

  vars:
    os_list:
      - ansible_distribution: "Amazon"
        ansible_distribution_version: "(Karoo)"
        ansible_distribution_major_version: "(Karoo)"
      - ansible_distribution: "Amazon"
        ansible_distribution_version: "2018.03"
        ansible_distribution_major_version: "2018"
      - ansible_distribution: "RedHat"
        ansible_distribution_version: "6.x"
        ansible_distribution_major_version: "6"
      - ansible_distribution: "RedHat"
        ansible_distribution_version: "7.x"
        ansible_distribution_major_version: "7"

  tasks:

    - name: Check conditions for recent distribs
      debug:
        msg: "This is a recent distrib"
      loop: "{{ os_list }}"
      loop_control:
        label: "{{ item.ansible_distribution }}-{{ item.ansible_distribution_version }}"
      when: (
              item.ansible_distribution == "Amazon"
              and
              item.ansible_distribution_version == "(Karoo)"
            )
            or
            (
              item.ansible_distribution == "RedHat"
              and
              item.ansible_distribution_major_version == "7"
            )

    - name: Check conditions for old distribs
      debug:
        msg: "This is an old distrib"
      loop: "{{ os_list }}"
      loop_control:
        label: "{{ item.ansible_distribution }}-{{ item.ansible_distribution_version }}"
      when: (
              item.ansible_distribution == "Amazon"
              and
              item.ansible_distribution_version == "2018.03"
            )
            or
            (
              item.ansible_distribution == "RedHat"
              and
              item.ansible_distribution_major_version == "6"
            )

And the result

PLAY [Statements for checking old/recent distribs] ************************************************************************************************************************************************************************

TASK [Check conditions for recent distribs] *******************************************************************************************************************************************************************************
ok: [localhost] => (item=Amazon-(Karoo)) => {
    "msg": "This is a recent distrib"
}
skipping: [localhost] => (item=Amazon-2018.03)
skipping: [localhost] => (item=RedHat-6.x)
ok: [localhost] => (item=RedHat-7.x) => {
    "msg": "This is a recent distrib"
}

TASK [Check conditions for old distribs] **********************************************************************************************************************************************************************************
skipping: [localhost] => (item=Amazon-(Karoo))
ok: [localhost] => (item=Amazon-2018.03) => {
    "msg": "This is an old distrib"
}
ok: [localhost] => (item=RedHat-6.x) => {
    "msg": "This is an old distrib"
}
skipping: [localhost] => (item=RedHat-7.x)

PLAY RECAP ****************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

As you can see it does the job as expected. So either your data is not gathered correctly or your check the wrong values.

Note: When @Vladimir Botka asks for an MCVE in the comments, this is a typical example.


Set container registry auth key via command line/env variable

You mentioned that the auth key is stored in Jenkins credentials and are available as environment variables, if so, then your half task is already done; what you can probably do from here is, use those environment variables to perform docker login (assuming USERNAME and PASSWORD are the Jenkins credentials variable), use the below command to login

docker login your.docker-registry.com -u $USERNAME -p $PASSWORD

This assumes, that your registry has SSL enabled, if so, don't forget to place the registry certificates under the below name on your Jenkin host

/etc/docker/certs.d/your.docker-registry/ca.crt

Post this, you would be able to push image to the registry


Using gitlab-ci to cache maven repository

To me the easiest route to this would be to have an instance of an artifactory running. There are several choices out there many of which have an open-source, community or free version that would support maven repositories.  Most options will support caching remote repositories (maven central for example) which can then be curated to enforce specific versions of the dependencies through your projects. It will also accept your own dependencies this way you can have a central unique access point for all maven dependencies in your team.

I've personally had success with Sonatype's Nexus (OSS version, or Pro version). There are other choices out there though check them out.

That said, if what you want is to prime the local repository for a new agent you want to provision, you could do so. But frankly, unless you have some real hefty dependencies (say in the GB size range), I would not bother doing so really and just let Maven naturally pull from the local binary repository instead. It makes for a simpler solution that will be more robust in the long run.


Ansible - merging inventories to share variables across?

While I haven't worked with plugin: aws_ec2, here's some general Ansible information that might help you:
The order of inventory files matter; the files given first will be read first; the ones read last will take precedence.  In your execution, you're referencing secrets.yaml last, so when aws_ec2.yaml is read, the secrets are still undefined.
You're trying to define a variable based on another variable with the same name.  That's asking for trouble; rename the reference variables, as that will help you debugging.
When defining variables for the inventory, it's generally better to not do that directly on the inventory itself.  See the link below on how to do that:
https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#organizing-host-and-group-variables
Finally, keyed_groups, regions and all these other keywords are specific to aws_ec2, and they'd work because you have plugin: aws_ec2 at the top of the first file.   You have no such definition on secrets.yaml, so Ansible would probably treat it as a normal inventory file, where that structure would not be accepted.  Your approach here is probably wrong, then.
Again, I never used that inventory plugin.  It may be the case that the inventory plugin does not allow for variables at that point.


Running parallel builds from jenkins pipeline job triggers and waits for same job on other project

Something needs to be different in other_job2 for tests['x'] and tests['y'].

Example(untested),

for (f in [ 'x', 'y' ]) { {
    {some code}
    build job: "other_job2", parameters: [file_params, string(name:'dummy', value: "${f}")], wait: true, propagate: true
    // the dummy parameter is for preventing mutation
}

More info:
https://www.jenkins.io/doc/pipeline/examples/#jobs-in-parallel


Azure service tags REST API and subscription level rights

To use servicetags api you have to register Microsoft.Netowork resource provider at azure subscription level.

Instead you can use these api:
to get the ip address list of  a specified Azure Service in a specified Azure region:
https://azinfo.azurewebsites.net/api/servicetags/{regionName}/{serviceName}
to get valid region names
https://azinfo.azurewebsites.net/api/GetRegionNameList
to get valid service names
https://azinfo.azurewebsites.net/api/GetServiceNameList
to get the all the ip address list of all Azure Services in a specified Azure region
https://azinfo.azurewebsites.net/api/region/{regionName}


What are best and comprehensive practices to consider when running docker in production?

The host on which the containers are running

Run the docker security bench on every node that runs docker containers https://github.com/docker/docker-bench-security

Running the following command on a node that runs docker containers:

docker run -it --net host --pid host --cap-add audit_control \
    -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

returns a list of checks:

[INFO] 1 - Host Configuration

[WARN] 1.1  - Ensure a separate partition for containers has been created

[NOTE] 4.2  - Ensure that containers use trusted base images

[PASS] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image

Quote from the repository README:

  The Docker Bench for Security is a script that checks for dozens of
  common best-practices around deploying Docker containers in
  production. The tests are all automated, and are inspired by the CIS
  Docker Community Edition Benchmark
  v1.1.0.

Some of the issues that are reported by the security bench could be solved by reading the official docker security article and comparing it with the bullets that are defined in the question the following things are important as well:

protect the docker daemon socket by implementing ssl
content trust using notary and DOCKER_CONTENT_TRUST variable


Amazon EKS from local kubectl does not cache MFA token/session

Indeed, this is a pain. I've been fighting with it for the whole day yesterday, but managed to get it working, let's see if our cases are similar enough.

My guess is that the usual aws configure configuration with a MFA ARN + the AWS SDK used to develop kubectl are not entirely compatible: I have two profiles set for this, one that has its mfa_serial_arn and role_arn that works as follows:

First aws command I type, I get prompted for MFA token
From there onwards, while session is active, I can work without typing in more tokens.
kubectl asks for a token EVERY SINGLE TIME.

This is bad.

Now, I set a different profile, without any of the above, and I do the following:

```

keys=($(aws sts assume-role --role-arn <ARN_OF_DELEGATED_ROLE> \
   --role-session-name  <RANDOM_UNIQUE_NAME> \
   --serial-number <ARN_OF_MFA_DEVICE>\
   --token-code  \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text))

# profile used here will be configured in ~/.aws/credentials
export AWS_PROFILE=<PROFILE YOU WANT>

aws configure set aws_access_key_id ${temp_keys[0]}
aws configure set aws_secret_access_key ${temp_keys[1]}
aws configure set aws_session_token ${temp_keys[2]}

```

This creates a temporary valid aws profile, that can be used BOTH with the aws cli, and kubectl.


Azure managed NFS shared storage for Azure VMs

You can use Azure Files
https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction
Limits for it can be found from here
https://docs.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets


How to rebuild multiple jobs at the same time in Jenkins?

You can use a pipeline to tie the jobs together, executing them either in parallel or sequentially through steps.
You can pass the relevant parameters to each job as you call it from the pipeline script.
Then you just need to rebuild a single pipeline job. You can tie the jobs together with a common build name through a variable as well.

build job: 'buildjob1', parameters: [
  [$class: 'StringParameterValue', name: 'BUILD_NAME', value: ${env.JOB_BASE_NAME}-${env.BUILD_NUMBER}"],
  [$class: 'StringParameterValue', name: 'FIXEDPARAM', value: 'some-string'],
  [$class: 'StringParameterValue', name: 'PARAM1', value: "${PARAM1}"]
]


What is the difference between a node and a machine on OpenShift?

Your interpretation is correct:

  My current understanding is that a machine represents a raw physical or virtual machine; just the computational resources like cpu, memory and storage. Nodes, then, would represent a machine that is configured for using with OpenShift, that is, a machine that contains the services necessary to run pods and is managed by the master components.

The differences arise on the level of abstraction.

A Node in OpenShift is exactly the same thing as a Kubernetes node. It refers to the abstraction of a worker/master "Node" in a "Cluster". The Node abstraction only exists within the boundaries of a running cluster.
A Machine is an abstraction introduced by the Cluster API which is an ongoing effort to allow deploying K8S clusters using a declarative approach. OpenShift is currently adopting the Cluster API hence they expect you to define the host infrastructure required to run your Nodes using the Machine spec.


Jenkins Pipeline - input only if branch matches

I see two approaches  :

Save current branch name on a variable and check it in a if statement

CURRENT_BRANCH = '...' // some value or parameter

stage('only on master') {
  if (CURRENT_BRANCH == 'master') {
  input {
    ...
  }
}

// be sure to check if input was set or define a default value

or

Run a shell script to know if current folder is a git repo on master

The following command check if current folder is a git repository  printing "true" ( or returning a error if does not have repo)

git rev-parse --is-inside-work-tree

The next, print the current branch name

git branch | grep \* | cut -d ' ' -f2

Use both to create a script that sets an environment variable.
Check the value on your pipeline, to show the input

stage('only on master') {

  // Set true on IS_ON_MASTER if current branch is master,
  // otherwise set false

  sh ./check-branch-master.sh

  if (env.IS_ON_MASTER) {
  input {
    ...
  }
}


Referencing locally built Docker images in Terraform

Yes, As we write terraform template is correct.
When you use terraform apply it will check local docker image list if it doesn't exit it will go to find registry as you define docker image.

If you don't want a pull image from the internet you can try the below steps. But at least you should have a base docker image of any Linux distribution.

Create a Dockerfile.
Build Dockerfile as below command

# docker build -t image_name -f Dockerfile

Define the docker image as a template

resource "docker_container" "my_service" {
  image = "image_name"
  name  = "my-service"
}

By terraform init install terraform module and apply the terraform template.

# terraform init

# terraform apply -auto-approve


https://forums.rancher.com/t/start-order-of-stack-containers/3106/9

  We do not support depends_on, and neither does Docker in Swarm mode.
  It is not a real solution to the problem anyway and leaves you with
  unhandled pointy-edge cases when failures occur and containers are
  being replaced. Your services should know how to either wait for their
  dependencies or exit and be rescheduled to retry.

I agree with this comment. Services should always be able to start. Imagine that one service goes down then another service will not work anymore.


Is there a better model for my CI/CD cycle?

Key issue with this approach is that applying commits selectively is extremely problematic. Basically, how do you know whether commits are dependent or independent?
As you correctly mentioned, this is one of the issues with Gitflow itself, so the solution is shifting methodologies. Preferred way today is Trunk-Based Development (TBD) - https://trunkbaseddevelopment.com/ - refer to "Accelerate" book for evidence.
Under TBD you can choose a style of work where you have short-lived feature branches. Once a feature is complete, pull request is created and QA can test this feature selectively under such short-lived branch and then approve the merge to main.
This way you know that your branches are independent (unlike commits in Gitflow) and approve features selectively. Note, that you can achieve something similar in Gitflow with multiple different branches, but this gets extremely messy there.


Multiple builds from one branch but different CSS values

If I correct understand Your problem, You want to build app with diffrent ccs and images.

Jenkins suports maven and simplest way is to use maven profiles plus maven-war-plugin
eg.:

<build>
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <configuration>
            <webResources>
                <webResource>
                    <directory>src/main/profile/webapp/${images.folder}</directory>
                    <targetPath>images</targetPath>
                </webResource>
            </webResources>
        </configuration>
    </plugin>
</plugins>

<profiles>
<profile>
    <id>prod_client_1</id>
    <properties>
        <images.folder>prod_client_1</images.folder>
    </properties>
</profile>
<profile>
    <id>prod_client_2</id>
    <properties>
        <images.folder>prod_client_2</images.folder>
    </properties>
</profile>

For other resources filtering maven-resources-plugin

At least should be helpful this approach sharing resources across projects


How to automatically publish git releases/comments to the site?

We did have a similar situation and did try out a few options. Let me suggest them and you can choose if they suit your need.
We had Jira for User Story (requirements), Bitbucket for SCM, Jenkins for CI orchestration, to name a few. Am not going into the other components since they may not be relevant from what I understand based on the description.
So, to start with, Jira would hold our (developer) user stories, and the developers start analysis based on that. Once they are clear, they pull the relevant code base from Bitbucket straight into the IDE (Eclipse) using git plugins, complete the work and push code back into Bitbucket. Two things were needed to be followed 'as part of the comment given during commit':

The Jira ID for the user story for which the change is being made
The actual comments for the change
E.g., [JIR-BBA]:: Code updated to pull correct URL from database.

Then, in the Jenkins job, we strip down the same into 'JIR-BBA', i.e. Jira ID, and the comment. The Jira ID would then be fed into the Bitbucket plugin which would directly update the corresponding Jira story. And this would also provide a hyperlink in Bitbucket which would directly take you to the job run in Jenkins, thus providing a 2-way communication.
Now the other part in your question, about the status of the build. For this we followed the below steps:

Created a Python script that would connect to Jira and can be run from the command prompt. This is easily available if you search for those 2 terms. The script would take in 4 parameters: i. Jira URL, ii. Jira credential, iii. Jira ID, iv. Comments
In Jenkins, we called the python script which would connect to Jira using the above credentials, and update the correct Jira task with the comments the developer provided (from the build job) and the job completion status (from SIT Deploy, SIT Test, UAT Deploy, etc.)
Now, inside Jira, once you go to the task, you shall find comments being updated for each of the jobs like 'Dev done', 'QA done', etc. depending on how you would want to get it update for each of the Jenkins jobs.
That way, you are maintaining a traceability for each of the requirements so that you know which stage each of them are deployed into or test failed.
Thus you do not need to go into Jenkins each time for the status as the latest is already available in Jira.


You can take a look at kube-monkey. According to the README, it randomly causes failures in kubernetes clusters.


Why can the certificate not be found by the Ingress Controller, while the certificate has been issued?

https://github.com/jetstack/cert-manager/issues/482#issuecomment-449664935

  Once I followed this
  https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/quick-start/index.rst#step-6---configure-lets-encrypt-issuer
  and deployed the issuers in the namespace of the app, it works


Just a quick follow-up on pumba. Installation (for Ubuntu):

$ curl -SL https://github.com/alexei-led/pumba/releases/download/0.7.2/pumba_linux_amd64 -O
$ sudo mv pumba_linux_amd64 /usr/bin/pumba
$ sudo chmod +x /usr/bin/pumba
$ pumba --version

It uses stress-ng under the hood, so same commands apply. Examples:

$ pumba stress -d 1m container_name

-d 1m - duration of a stress test (1 minute)

container_name - your target container (from docker ps output)
$ pumba stress -d 30s --stressors "--vm 10 --vm-bytes 512M --vm-hang 20" container_name

-d 30 - duration 30 seconds

-stressors - parameters passed to stress-ng app

In this particular example we create 10 workers spinning on malloc()/free() allocating 512MB each and sleep 20 seconds before freeing the memory.

Default stressor as in first example is --cpu 4 --timeout 60s


Trying to make job working AWS MediaConvert (ERROR 1040)

I got this exact same error. I had to add the Audio Selectors. See the following JSON example:

"Inputs": [
  {
    "AudioSelectors": {
      "Audio Selector 1": {
        "Offset": 0,
        "DefaultSelection": "DEFAULT",
        "SelectorType": "LANGUAGE_CODE",
        "ProgramSelection": 1,
        "LanguageCode": "ENM"
      }
    },

If you are using the Java SDK, add something similar to the following to your inputs:

AudioSelector audioSelector = new AudioSelector()
            .withOffset(0)
            .withDefaultSelection("DEFAULT")
            .withLanguageCode("ENM")
            .withSelectorType("LANGUAGE_CODE")
            .withProgramSelection(1);

audioSelectors.put("Audio Selector 1", audioSelector);


How do I trigger scripts on a merge or when a branch is deleted?

"This is great, but I want to run a command/script when a branch is removed, how can I have jenkins run a command when a branch is removed?"

  You will need to use a webhook for this, I use the following
  settings in Github:

  Bitbucket only supports the following webhooks:

"Is there a way to have Jenkins trigger a job before it removes the branch?"

  Not that I can see, but in Github you can create a custom label and
  trigger a job via a webhook when the label is added to a branch.


Terraform ignore_changes block on JSON-encoded attribute

Due to how that resource type is designed, from Terraform's standpoint replication_task_settings is just an opaque string and so Terraform can either detect that the string has changed (the default) or ignore changes to the string as a whole.
There is no way to ignore changes to only part of a string, because Terraform implements ignore_changes by taking the previous value of that argument instead of the new value. Terraform doesn't and can't know how to construct an new string that combines some characters from the old string and some characters from the new string, because it has no awareness of the meaning of that string.


What are possible backup status in velero?

As I see in Velero code (version: v.1.5.3), the backups can have the following phases:

New
FailedValidation
InProgress
Completed
PartiallyFailed
Failed
Deleting

see: https://github.com/vmware-tanzu/velero/blob/v1.5.3/pkg/apis/velero/v1/backup.go#L195
Restores have these phases:

New
FailedValidation
InProgress
Completed
PartiallyFailed
Failed

see: https://github.com/vmware-tanzu/velero/blob/v1.5.3/pkg/apis/velero/v1/restore.go#L184


Should worker processes of web applications be placed in separate containers?

I can think of only one case where overhead of running workers in separate containers is justified: if your setup uses docker swarm for clustered deployment. This way you will get all of the HA benefits.

Otherwise I don't see a reason to complicate such tasks, especially if they must use strictly the same codebase (which leads me to believe that they share one domain of concerns, so to say). It might seem elegant, but I just don't see enough benefits

p.s. any split of such kind will force you to manage interconnection and complicate setup for local development


Travis tests for R package fail with little explanation

If you are using the recommended setting for R packages of upgrading warnings to errors, i.e. with this line in .travis.yml:

warnings_are_errors: true

then the place to look is the previous heading in the Travis log, called Checking package.

Example of problem with missing font

In my case, the Travis log under the heading Checking package showed that the
inconsolata font was missing, so the vignettes failed to build:

* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
LaTeX errors found:
! LaTeX Error: File `inconsolata.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
! Emergency stop.
<read *>

l.276 ^^M

!  ==> Fatal error occurred, no output PDF file produced!
* checking PDF version of manual without hyperrefs or index ... ERROR
* DONE
Status: 1 ERROR, 1 WARNING

This caused an error for LaTeX, a warning for R, and an error for Travis.

Solution to problem

In my case, the problem was due to Travis using TeXLive 2019 and the CTAN
mirrors having yet to update to 2019.

The issue appears in the Travis R community forum:

  might be related to the recent switch to TexLive-2019 as the LaTeX package manager seems to fail finding inconsolata in the repository.

  ...

  It looks to me like the mirror chosen was has not updated to 2019, so I think this will resolve itself naturally when the mirrors catch up. If not I can look into it.

In the meantime, I solved it by forcing a previous version of TeXLive with a downgrade to OSX 10.12 in .travis.yml:

# macOS 10.12 version
osx_image: xcode9.2

See this Travis
page for the
correspondence between XCode and OSX versions and the original thread in DevOps
StackExchange.


Have a look at the documentation presented here:

https://github.com/jenkinsci/kubernetes-plugin

You need to specify a Kubernetes pod in your pipeline. An example of your use case would look similar to this for a single python container. You'll notice that containers: [] expects an array so you can include additional containers.

def label = "mypod-${UUID.randomUUID().toString()}"

podTemplate(label: label, containers: [
    containerTemplate(name: 'python', image: 'python:3.7-alpine', ttyEnabled: true, command: 'cat'),
]) {
    node(label) {
        stage('Get a Python project') {
            git 'https://github.com/oğuz/project.git'
            container('python') {
                stage('Test a Python project') {
                    sh "python -m -v py.test TEST_FILE.py "
                }
            }
        }
    }
}


If you just want to print a subset of the results of your commands on your terminal without using one or more -v flags, you can refactor your playbook to show just the standard output from each of the commands using a JSON query. For example:
---
- name: display info
  hosts: localhost
  tasks:
  - name: Display uptime and kernel info
    shell: "{{ item }}"
    with_items:
      - uptime
      - uname -n
      - uname -r
    register: output

  - debug: msg="{{ output.results | json_query('[].stdout[]') }}"

This will yield output similar to the following:
PLAY [display info] **********************************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [localhost]

TASK [Display uptime and kernel info] ****************************************************************
changed: [localhost] => (item=uptime)
changed: [localhost] => (item=uname -n)
changed: [localhost] => (item=uname -r)

TASK [debug] *****************************************************************************************
ok: [localhost] => {
    "msg": [
        "22:48  up 4 days,  3:41, 2 users, load averages: 1.26 1.39 1.42",
        "localhost.local",
        "20.3.0"
    ]
}

PLAY RECAP *******************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


How to exclude storage.bucket.list permission on public gcp storage bucket

I am not entirely sure what configs you were using, but here is a process using CLI (gcloud and gsutil). The end product is a new custom role replacing the current standard roles/storage.objectViewer on allUsers for a GCP Bucket.
Recommend using GCP Cloud Cloud Shell in your Project: https://cloud.google.com/shell

Create a new Role with the Custom Permissions you would like.

Create yaml file to hold the new config:
vi custom_role.yaml

Add role configurations to new custom_role.yaml:
title: "storageobjectviewer.nolist"
description: "Storage Object Viewer Role without source objects list"
stage: "ALPHA"
includedPermissions:
- resourcemanager.projects.get
- storage.objects.get

Create new Role in GCP:
gcloud iam roles create storageobjectviewer.nolist --project=[YOUR PROJECT] --file=custom_role.yaml

Pull current permissions for [YOUR BUCKET] locally:

gsutil iam get gs://[YOUR BUCKET] > perms.txt

Update Permissions with the new role.

Should only be this section:

{
  "members": [
    "allUsers"
  ],
  "role": "projects/[YOUR PROJECT]/roles/storageobjectviewer.nolist"
}

Update the allUsers to the new role on [YOUR BUCKET]

gsutil iam set perms.txt  gs://[YOUR BUCKET]

The new configuration should be viewable in CLI:
gsutil iam get gs://[YOUR BUCKET]

Also in the UI:
Bucket Details -> Permissions

and IAM -> Roles

Hopefully this helps!
Additional Info:

https://cloud.google.com/iam/docs/creating-custom-roles#iam-custom-roles-create-gcloud
https://cloud.google.com/storage/docs/gsutil/commands/iam
https://cloud.google.com/storage/docs/access-control/making-data-public#gsutil
https://cloud.google.com/storage/docs/access-control/iam-roles


Which S3 buckets is Lambda allowed to read code from?

This represents a misunderstanding of when Lambda code is loaded.

AWS Lambda does not download a copy of the code from S3 every time a function is executed. Instead, Lambda has its own internal storage that is updated when invoking CreateFunction or UpdateFunctionCode either through the API or the Console.

Update with further clarification from jdog:

  Lambda is never given access to S3 buckets -- the user or role that runs CreateFunction is.


The AWS answer would be to use appropriate services for each component.
E.g. Put the database on an RDS - they have multiple choices. Deploy the front-end using Fargate and Lambda.
There are 2 issues to consider:
1. The learning curve.
Getting up to speed on AWS technology - and keeping up with them as they make minor tweaks that break lagacy deployments - can get very expensive, time-wise.
OTOH, once you're an expert at these technologies, you can leverage them with amazing results.
2. The costs:
While you develop you will be using the free-tier mostly, and the costs will be minimal.
Once you move into production, you have have to start using "real" machinery, and even if you don't, most services have an expiration for their free tier: 12 months and/or a certain amount of usage.
At that point the costs start ballooning. from small-change to 3 or even 4-figure numbers for a small installation.

The other approach is to do it the way you'd do it without AWS.
You didn't give me much details, but if you're deploying something simple, you may want to spin up an EC2 instance, install a LAMP server and use it for both front & back end.
You can then use the Lifecycle Manager to back up the entire machine daily.
However, if you have to scale this up in a big way, you'll have to start from scratch.

So, what do you choose?
The answer depends on your resources: time, money and manpower, as well as how you see this growing from a one-off project to a long term heavily used installation that needs redundancy, load balancing and the ability to scale dynamically.
You could also look at something like AWS Beanstalk, which is easier to implement and redeploy, but not as dynamic, as their serverless approach.


Does Oracle have an API that shows the latest stable JDK version?

http://api.releasesoftwaremoreoften.com/latestversion?name=oraclejdk8
http://api.releasesoftwaremoreoften.com/latestversion?name=oraclejdk9

Running:

curl -s http://releasesoftwaremoreoften.com/latestversion?name=oraclejdk9 | jq .latestVersion

returns:

9.0.4

and

curl -s http://releasesoftwaremoreoften.com/latestversion?name=oraclejdk8 | jq .latestVersion

returns:

8u162


Install MDB2 in docker container while PEAR is down

You need to learn a bit more about Debian or Ubuntu. If you look at every single Debian or Ubuntu container that installs things, you will notice, amongst other things:

apt-get update && apt -f install [whatever]

The reason for this is that aptitude doesn't know where to find anything without the "apt update".

See for example  https://github.com/dockerfile/ubuntu/blob/master/Dockerfile

# Install.
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update && \
  apt-get -y upgrade && \
  apt-get install -y build-essential && \
  apt-get install -y software-properties-common && \
  apt-get install -y byobu curl git htop man unzip vim wget && \
  rm -rf /var/lib/apt/lists/*

This updates all packages, then installs build-essential, then software-properties-common and then the packages in the second to last line. Of course you don't need all of this.

It is important however to have the apt-get update && apt-get install on the same line, if you are ever going to deploy your Dockerfile to any CI system. If apt-get update is on its own line, then that layer will never get invalidated as the succeeding layers will have old sources that they build consequent layers from. Keep in mind that every single command generates its own layer, that is why commands are chained in this way - so as to create one layer instead of many that will never serve any purpose.


I believe this is due to non existing variable attribute.
Most likely chain_from_certificate1 and chain_from_certificate2 don't have stdout
Did you try debug by printing the variables?
With stdout:
Did this throw similar failure?
    - name: print variable
      debug:
        msg: "{{ chain_from_certificate1.stdout }}"

Without stdout:
What is output of this?
Will this print same output as of manually execute openssl x509 -issuer -noout -in Certiticate1?
    - name: print variable
      debug:
        msg: "{{ chain_from_certificate1 }}"

My assumption is that the script set the value of stdout in chain_from_certificate1 and chain_from_certificate2 rather than the stdout: <value>. Therefor stdout do not exist as attribute.
Or chain_from_certificate1 and chain_from_certificate2 contain another data structure.
The key is to print them out.


Terraform: how do you inspect sensitive data?

When your configuration or a provider marks an attribute as sensitive, Terraform will always hide that value in any output that's intended for human consumption.
The real values are available in machine-readable output though. This is primarily with the aim of integrating with external software, but if you need to then you can also inspect the machine-readable output directly yourself.
You can get a machine-readable (JSON) rendering of a plan like this:
terraform plan -out=tfplan
terraform show -json tfplan

If you intend to read it directly in the terminal then it can help to pipe it into jq, if you have that utility installed:
terraform show -json tfplan | jq


Docker and Kubernetes integrations compatibility matrix?

Every Kubernetes release has an External Dependencies section in the respective Changelog. E.g.:

CHANGELOG-1.14:

  The list of validated docker versions has changed. 1.11.1 and 1.12.1
  have been removed. The current list is 1.13.1, 17.03, 17.06, 17.09,
  18.06, 18.09.

CHANGELOG-1.13:

  The list of validated docker versions remain unchanged at 1.11.1,
  1.12.1, 1.13.1, 17.03, 17.06, 17.09, 18.06 since Kubernetes 1.12.

CHANGELOG-1.12:

  The list of validated docker versions was updated to 1.11.1, 1.12.1,
  1.13.1, 17.03, 17.06, 17.09, 18.06.

and so on...


Jenkins Credentials Manager + Active Choices

It sounds to me like you're already aware of this, but there are a couple easy ways to do this, and a couple hard ways to do this.
The easy ways:

Make your repo available for checkout without SSH keys or authentication credentials.
Manually add an SSH keypair for repo checkout to the Jenkins user on your Jenkins master node (since Active Choice Groovy scripts run on the master node) (i.e. don't use the Credentials store).

The hard ways:

Use the Jenkins Java API in your Active Choices code to extract credentials from the credentials store.  API docs are available for the credentials store and the ssh credentials type.  There is no documentation on how to use the Java classes available in these plugins, so it's up to you to figure it out yourself.
Create a secondary "meta" job to update the parameters in the original Jenkins job (including Active Choices scripts) - I describe this a bit below.

If you can't do any of the easy solutions for security reasons, there isn't an easy, straightforward way to update Active Choice parameters based on the contents of a git repo.  This is because:

Like you said, "during pre-pipeline stage (i.e. Active Choices parameter) Hudson extensions such as withCredentials or sshAgent are not accessible".
The configuration for parameters, including Active Choice parameters, is taken from the previous run of that Pipeline, not the current one.  Meaning that, to update your parameters from within your Pipeline code, you'll need to run your Pipeline, which defeats the point of what you want to do.

There is a solution for 1., which is to use Scripted Pipelines instead of Declarative, but that won't get around the problems of 2.
The solution that I came up with when I ran into this issue is to create a second Jenkins job (I call it a "meta" job) which updates the parameters of the original job, including Active Choices scripts.  This second meta job is triggered by a git hook on the repository that I use to fill the options in my Active Choices parameters.  This has the advantage that you do not need to run the original job's Pipeline in order to update its parameters, solving the issues in 2.  Also, because it's still a Pipeline job, you can use normal Pipeline steps, thus allowing you to access the credentials store.
My job looks (roughly) like this - I simplified it a lot to make it fit here, so you'll need to do a bunch of tweaking to get it working the way you want.  There are probably a ton of errors and bugs in this because I ripped apart an existing script to make this.  A few things to note:

This makes heavy use of Jenkins APIs, so you'll need to either disable the script sandbox or whitelist a ton of functions in the Jenkins security manager in order to get this to run
This is still technically a Scripted Pipeline even though it looks much more like a normal Groovy script than a Pipeline file
Because this is a Pipeline, you can use steps like withCredentials to check out repositories using the credentials store.

import groovy.json.JsonOutput

import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript

import org.biouno.unochoice.*
import org.biouno.unochoice.model.*

def metadataUrl = 'ssh://git@**************/metadata.git'
def metadataBranch = 'master'

def metadata = [:]

node('master') {
  git(
    url: metadataUrl,
    branch: metadataBranch,
    credentialsId: 'jenkins',
    changelog: false,
  )

  def dataDir = "${pwd()}/myJsonFiles"

  // can't use File.eachFileMatch() due to:
  // https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches
  def files = new File(dataDir).list().findAll() { f->
    FilenameUtils.getExtension(f) == 'json'
  }

  files.each() { f->
    // TODO: nice error messages if files are misformatted
    def name = FilenameUtils.removeExtension(f)
    def data = readJson(file: "${dataDir}/${f}")
    metadata[name] = data
  }
}

// ChoiceParameterDefinition is the class name for the built-in static choice parameter.
// CascadeChoiceParameter is the class name for the Active Choices dynamic choice parameter.

// begin first parameter
def fileParam = new ChoiceParameterDefinition(
  'file',
  'The JSON file to use',
)
fileParam.setChoices(metadata.keySet())
// end first parameter

// begin second parameter
def metadataJson = JsonOutput.toJson(metadata)

def scriptText = """
import groovy.json.JsonSlurper

def slurper = new JsonSlurper()
def metadata = slurper.parseText('${metadataJson}')

return metadata[file]['versions']
"""
def mainScript = new SecureGroovyScript(scriptText, true)
def fallbackScript = new SecureGroovyScript('return ["Error encountered - see logs"]', true)
def activeChoiceScript = new GroovyScript(mainScript, fallbackScript)

def versionParam = new CascadeChoiceParameter(
  'version',
  'The version to use',
  activeChoiceScript,
  'PT_SINGLE_SELECT',
  'file',
  false,
)
// end second parameter

def paramProp = new ParametersDefinitionProperty(fileParam, versionParam)

// this is the full name of the actual job which needs dynamic parameters
def job = Jenkins.instance.getItemByFullName('UNIX/deploy_to_environment/master')

// There is no "setProperty" - we need to remove and replace.
// removeProperty is safe to run even if no matching properties are set on the job.
// However, this method only removes one property per method call.
// Theoretically, this means the job could accumulate multiple conflicting
// ParametersDefinitionProperty properties.
// TODO: run removeProperty in a while loop to eliminate all matching properties for extra safety.
job.removeProperty(ParametersDefinitionProperty)

job.addProperty(paramProp)


Additional push event syncing between Artifactory instances

Yes, you can set up push event-driven replication. Please note that while pull replication prepopulates remote repository in server A from a local repository in server B, push replication replicates between two local repositories. Once you set up a virtual repository it doesn't really matter because you'll have a single point of access anyway, but just something for you to be aware of.

You can read more about push replication and pull replication in the user guide. Also, please find here a whitepaper on different replication topologies and a webinar on the same topic.


What is the proper way to fix `error: did not detect an --insecure-registry argument on the Docker daemon` while running `oc cluster up`?

This error happens because the oc cluster up command spawns a Docker registry. The 172.30.0.0/16 IP range is probably corresponding to your Docker bridge network. As the registry will run in a container with an unpredictable IP address, you must indicate the IP range (--insecure-registry 172.30.0.0/16).


How to properly achieve dynamic parallel action with a declarative pipeline?

Managed to solve it with the following code:

pipeline {
    agent { label "master"}
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }
    }
}


Kubernetes flannel pod getting the wrong network

You have either a bad understanding or misconception about how a cidr works.

10.105.5.128/21 is just one address in the range 10.105.0.0 - 10.105.7.255.
So either your cidr bits are not the one you're expecting, 10.105.5.128/26 would be a valid network definition but that's a /26 subnet within the 10.105.0.0/21, or you're not using the proper network definition.

You can check those division on this very helping site (the link already use your networks above.


Testing R package on Travis fails because of TeXLive incompatible versions

This issue is due to the CTAN mirrors having yet to update. The issue appears in the Travis R community forum:

  might be related to the recent switch to TexLive-2019 as the LaTeX package manager seems to fail finding inconsolata in the repository.

  ...

  It looks to me like the mirror chosen was has not updated to 2019, so I think this will resolve itself naturally when the mirrors catch up. If not I can look into it.

In the meantime, I solved it by forcing a previous version of TeXLive with a downgrade to OSX 10.12 in .travis.yml:

# macOS 10.12 version
osx_image: xcode9.2

See this Travis page for the correspondence between XCode and OSX versions.


Is it a bad idea to use a web app including nginx, gunicorn and flask in a single docker container?

While this is possible it is not a best practice.  Whenever possible, containers should contain a single process.  The reasoning for this is:

It fosters a modular design where you can more easily plug and play components.  What if you decide not to use nginx and use asp.net core in the future?  If everything is tightly coupled in one container then the entire container will need to be reworked any time one of these components changes.  Look into microservices for more information on the benefits a modular design.
It is easier to parse container logs as each container will have its own distinct output.
If a process crashes and everything is in one container, the entire container will crash.  If you have them broken out across multiple containers, recovery is simplified.  While this may not be applicable for running standalone containers, it is very useful if the containers are running in a Kubernetes Pod as the Pod can restart.


docker.containers.create() is documented to:

  Create a container without starting it. Similar to docker create.

Either call docker.containers.run() instead, or given the container object returned from create(), call ingreedy.start().

(docker ps by default only shows running containers; docker ps -a should show the container in a "created" state.)


How to get mobile push notifications on Jenkins jobs?

Jenkins has a bunch of plugins:

There's HTML5 desktop notifier plugin which can push notifications to your browser
There's a Slack plugin which can push notifications to a Slack team/channel


How to deploy SQL Script in DevOps culture and Rollback situation too?

There are a few different things you can do:

In your CI process, run your new SQL update scripts against a Dockerized SQL Server. This will allow your CI server to reject the changes if there are basic errors in the script.
If you are fully containerized look into tools like Kubernetes.  These tools have ways to handle rolling back failed production updates.
Utilize a Blue Green Deployment strategy along with integration testing.  This provides a fallback in case things go wrong and allows for different types of automated testing schemes in your CI process.


AWS RDS- Determine what Database Tables aren't interacted with?

I don't think any database services "internal traffic" is available to AWS services directly.

That being said, you should be able to dump all sql queries to a log file on S3 and also extract all database/table combinations to a text file as well.

It then becomes a text processing exercise that you can possibly solve with AWS Athena


Setting up CI/CD for an AWS CDK app using AWS CodeBuild/Deploy/Pipeline

It may be helpful to look at the different AWS developer services as you are trying to achieve your goal in the wrong services:

Running the specific dotnet testing and publish steps are not done in CodeDeploy.  Rather, they would be done in AWS CodeBuild, in the AWS Pipeline, or directly on your Jenkins Server.
AWS CodeDeploy is for taking your build artifacts and deploying them "to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers".
AWS CodePipeline is for modeling an automated workflow.  This service integrates with Jenkins and GitHub.

With these services in mind I think what you are looking for is to start with a new CodePipeline.  This will allow you to start building a model that specifies where you are running your dotnet tests (Jenkins or on AWS) and what to do with your artifacts (publish through Jenkins or AWS CodeDeploy).  This can be accomplished with AWS CDK.


If the end result would be entirely contained in a directory somewhere you could simply archive that directory after the installation on one machine and restore it on the new machine. It does work for some apps.

But after going through the Ubuntu 18.04: How to install OpenCV procedure and peeking into the makefiles I believe this wouldn't be applicable to OpenCV: its artefacts are spread in various places on a Ubuntu system.

But I think it's possible to significantly reduce the installation time by skipping the actual build process, which appears to be the bulk of the process. From the above referenced procedure (which I'll also use in describing the proposed solution):

  This process may take 30 minutes or longer, so go for a nice walk if
  you are able.

I'll re-use the referenced procedure for describing the proposal, with a few tweaks (I'd like to think that the tweaks would easily be portable to future/similar versions of the procedure):

In the initial installation:

after the build completes but before the Installing and verifying OpenCV step insert:

Step 4a: save the pre-built official OpenCV source tree to S3

Follow Copying from s3 to EC2 instance instructions for S3 access setup, then run:

tar cvfz pre_built_opencv.tgz opencv*
s3cmd put pre_built_opencv.tgz s3://<bucket_name>

Optional: you can remove the local archive copy now:

rm pre_built_opencv.tgz

In the subsequent installations:

replace Step #2: Download the official OpenCV source with:

Step #2: Extract the pre-built official OpenCV source tree from S3

Follow Copying from s3 to EC2 instance instructions for S3 access setup, then run:

cd ~
s3cmd get s3://<bucket_name>/pre_built_opencv.tgz pre_built_opencv.tgz
tar xvzf pre_built_opencv.tgz

Optional: you can remove the local archive copy now:

rm pre_built_opencv.tgz

in Step #4: Configure and compile OpenCV for Ubuntu 18.04:

skip the Configure OpenCV with CMake section
replace the Compiling OpenCV on Ubuntu 18.04 section with:

Enter the pre-built OpenCV build directory

cd ~/opencv/build

I didn't run it myself, but I think it should work, at least for VMs of the same type.

I'd 1st try the same even for different VM types. If that doesn't work then I'd re-run the initial installation for them, using a different archive filename.


Connect external Jenkins Master to GKE with Kubernetes plugin

You could install Jenkins using helm to install the k8s plugin as well.

https://github.com/helm/charts/tree/master/stable/jenkins


Jenkins Declarative Pipeline: Can Jenkins execute a dynamic list of jobs?

I'm not really sure I understand your question, but you can execute arbitrary Groovy inside of a script block, so you can do just about anything within script.
For example, here is my interpretation of what you are asking for:
stage ('Execute jobs') {
    steps {
        node('node1') {
            script {
                parallel_jobs = [:]
                if (params.includeJob1 == "true") {
                    parallel_jobs['ci'] = {
                        build job: job1, parameters: [], propagate: true
                    }
                }
                parallel_jobs['second'] = {
                    build job: job2, parameters: [], propagate: true
                }

                // ...
                // and more jobs dinamically added to the parallel_jobs list
                // ... parallel_jobs['etc'] = ...

                if (params.parallel) {
                    // execute jobs in parallel
                    parallel(parallel_jobs)
                } else {
                    // execute jobs in sequence
                    parallel_jobs.each { name, closure->
                        closure.call()
                    }
                }
            }
        }
    }
}

In Groovy, Maps (i.e. associative arrays) are ordered.  So if you use this to execute the builds serially rather than in parallel, the builds will be executed in the order they are defined.


Why not run a stateless Hashicorp Vault in production?

I think you're misunderstanding what is considered bad practice.

Running vault with an in memory backend and being able to spin it up and provision it is fine. What is Not Fine is seeding the secrets in Vault from an insecure source.

If you are just using Vault as a way to simplify operations, as opposed to a long term persistent state store - then go for it.

One caveat - if you use the capabilities in vault to rotate keys for other integrated services then if you lose your vault cluster then you have just lost the keys to those services, so hopefully anything accessed using a Vault managed, regularly rotated, key is also easy for you to reprovision


Route53 has an API for creating DNS records.  The API is a bit obtuse, but there are a plethora of wrappers for various languages.


Deploy Docker Compose from different repositories

You can utilize watchtower to restart the specified containers each time new image of that container is avaialble.

Watchtower : with watchtower you can update the running version
of your containerized app simply by pushing a new image to the Docker
Hub or your own image registry. Watchtower will pull down your new
image, gracefully shut down your existing container and restart it
with the same options that were used when it was deployed initially.
Run the watchtower container with the following command:
$ docker run -d \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower

docker-compose integration example:
services:
  watchtower:
    command: --label-enable --cleanup --interval 300
    image: containrrr/watchtower
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    network_mode: none
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  mariadb:
      depends_on:
        - watchtower
      env_file: .mariadb.env
      image: mariadb:10
      labels:
        - "com.centurylinklabs.watchtower.enable=true"
      networks:
        - default
      ports:
        - 3306:3306
      restart: always
      volumes:
        - ./data/volumes/mariadb:/var/lib/mysql

To check the logs of the updation you can execute docker-compose logs watchtower
docker-compose integration reference: https://chriswiegman.com/2019/12/keeping-docker-containers-updated/


Which libraries generate observability metrics for applications in the cluster to prometheus?

For Application specific calls you need to use a Prometheus client and create the metrics yourself. The Prometheus client will allow you to create a /metrics endpoint where you can output the current metric values which Prometheus will scrape.

For metrics that are not unique to your application, you can use Prometheus exporters that will run on either the node, container, or runtime environment, or third-party software. These will also create targets which you can tell Prometheus to scrape to gather the metric data.

Once the metric data is collected you'll need to write your own RED and USE queries in graphana to display those metrics.


How to handle updating binary files on a dev server with git

Git doesn't consider the untracked files for making decisions about accepting or rejecting a git pull operation, so you should have no worry about it.

If still in doubt you can easily verify it: just pull another copy of the same git repo in some other location - for test only - mess with untracked files in it and pull again.


How to run a CMD in a docker container that was created using Packer?

And then I start like this docker run -p 21:21 -it --rm 52c0b0362362 bash

This would not work by design. When you run the image with the command above you are instructing Docker to overwrite the CMD instruction from your Dockerfile and therefore telling it to execute bash instead of sudo myprogram. It doesn't matter what you have set in the CMD instruction - it would never be executed.

One way you could go around this would be to use ENTRYPOINT as others have suggested. This will make sudo myprogram the default command and then CMD can act as "additional arguments" to the default command (sudo myprogram).

Here's a working example:

# build.json
{
    "builders": [
        {
            "type": "docker",
            "image": "ubuntu:latest",
            "commit": true,
            "changes": [
                "ONBUILD RUN apt-get -q update",
                "ONBUILD RUN apt-get install -y -q htop",
                "ENTRYPOINT [\"top\"]"
            ]
        }
    ]
}

# /packer build ./build.json

# run top with delay 3 sec
# docker run --rm -it sha256:hash

# run top with delay 1 sec
# docker run --rm -it sha256:hash -d 1

The other solution would be to keep the CMD [\"sudo myprogram\"] instruction but to run the image without adding a command at the end: docker run -it --rm HASH_ID.


Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that train_model_config_A, train_model_config_B, etc.

Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding train_model_config_<blah> script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.

You might not need to install the slave directly on the GPU cluster if the train_model_config_<blah> scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.


Can I change the mount path and port number of a built container?

Containers are designed to be disposable. Delete the "A" container and recreate it with the new volume mount and port number. Note that deleting the container will delete any changes made within the container specific RW filesystem layers. If you have changes in there that you need to preserve, then you really want to make those volume mounts. In the end, the workflow looks like:

docker container rm ${old_container_id}
docker container run -v ${new_volume} -p ${new_port} ...

As for changing an existing container, there is:

docker container update ...

however, changing volumes and port bindings are not part of the settings designed to be changed from this. Going under the hood of the docker engine to try to change these settings manually is unsupported and may result in unexpected behavior or a broken engine.


You could extract the common, shareable portions of the scripts which you'd package as either standalone scripts or libraries (depending on the scripting language you use and/or preferences), placed in a shared location (of course, backed by their own repository).

Then, in the scripts from your code repositories you'd replace the extracted portions with invocations of the scripts (or calls into the libraries) from the shared location.

Some CI/CD tools might be capable of directly invoking the scripts from the shared location, so if there aren't any specific portions in some of the code repo scripts you could drop them completely from those repos.


Trigger gitlab downstream pipeline from a upstream pipeline in a multi-project using trigger, rules and custom configuration variables

You could use dynamic child pipelines. (https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines).
You could create a job for triggering your job "buildArtifact" in a yml-file created with jsonnet in your "checkArtifactPresent".
With using "--ext-str" or "--ext-code" in jsonnet.
You can control the yml-file creation (https://jsonnet.org/learning/tutorial.html#parameterize-entire-config).
Read more https://jsonnet.org/


How to get the ip range of github-production-release-asset-2e65be.s3.amazonaws.com?

@Tensibai indicated that:

  See: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html if you
  really want to go this way, but there's quite large ranges, the
  52.216.0.0 is a /15 for us-east-1 and not a /16 btw { "ip_prefix": "52.216.0.0/15", "region": "us-east-1", "service": "S3" },


How do I include my internal CA certificate to validate SSL exchanges in Chef?

There's a couple of way to achieve the result:

Chef has a trusted_dir to allow adding certificate to the trusted list. the documentation has a lot of details about it. Adding your CA certificate to this directory would solve the problem. knife has it also in a slightly different path as per it's own documentation
Chef use its own CA certiticate list in /opt/chef/embedded/ssl/certs/cacert.pem.
You can add your CA certificate at end of this list to trust it.

The second option has an advantage of allowing you to export the environment variable SSL_CERT_FILE pointing to chef cacert.pem to allow most of the tools using openssl library to know your CA certificate.

For the case of a self signed certificate on the chef server (or another server used as target in a recipe), knife ssl_fetch would allow all knife commands to work.

To add the server certificate to the cacert.pem for the case 2. above, you can use the following command:

# For a self signed CA certiticate
openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >> /opt/chefdk/embedded/ssl/certs/cacert.pem

# For an internal CA signed certificate:
openssl s_client -showcerts -verify 5 -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null | awk '/BEGIN/,/END/{if(/BEGIN/){a++}; certs[a]=(certs[a] "\n" $0)}; END {print certs[a]}' >> /opt/chefdk/embedded/ssl/certs/cacert.pem

export SSL_CERT_FILE=/opt/chefdk/embedded/ssl/certs/cacert.pem

The openssl command is included in chef-dk, so this can be done under windows also.  Change the path to c:\opscode\ instead of /opt/. To export the environment variable use set SSL_CERT_FILE=... (with /P to add it permanently to your environment) in your command.


I got a work around to this case. basically, My root volume is 100GB and I created a separate partition out of it as 70GB and I set it to a new device sda3 and mounted it to /var/lib.
now, all docker container logs are written in same path but in backend they are mounted to a different drive. with this setup, even if there is disk pressure, the node does not become unavailable.
so my drive now looks liek below
sda1
sda2 / 30GB
sda3 /var/lib 70GB.  << new device


How to find the image name corresponding to a layer digest?

The mapping between image manifest and layers goes the other way. You can have one or more image manifests pointing to a layer. And those images may not even be tagged with an image name, or a single image may have multiple tags. So I'm not aware of an easy way to do a reverse search. You would need to go through each image on the local system or registry to see what images point to that layer, and then search the tags to see what names that image has.


How can I convince someone that 100% reliability is not the right target for anything?

I was wondering whether anyone in this forum had new arguments they have heard or tried in person to convince executives of the need to set service level objectives at a level below 100%, hence the original question.

Here are some approaches I've seen or taken in the past to tackling this problem.  The Site Reliability Workbook has a few suggestions here, which include:

100% is not a reasonable target since every component of any system has a
non-zero probability of failure.
External components (such as an ISP) sitting between the customer and the target system aren't 100% reliable.
100% reliability implies you can never change the system, since all change introduces risk.
100% reliability implies you spend all your time reacting to reliability problems and have no time for anything else.

In addition to the above, I've also gathered the following talking points from various in-person discussions that might be useful to persuade organizational leaders:

If an IT exec sets a target of 100% reliability, they are basically encouraging their engineers to lie to them and hide problems, which means they will discover those problems in the most expensive possible way.
Nature has never evolved a 100% reliable system. Gene replication is unreliable. The human heart is not 100% reliable.  Life continues and thrives in the face of imperfect reliability, and so can technology systems and their users.
No group of humans has ever engineered a system that's 100% reliable in the long run.

We also have to acknowledge that for certain dimensions of reliability, such as data loss prevention, some regulated industries require levels of reliability so high and over long periods of time that they are virtually indistinguishable from 100%, like 99.999999%. For other dimensions like availability and latency, three or four nines of reliability are often more than adequate, and some of the arguments above may help persuade IT leadership that this is true.


I found exactly what I was looking for-

[Jenkins plugin] (https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin)

Which let you create a lock between, slaves , different jobs and more..

(in contrast to the plugin description)


Docker's value and scenarios in native Windows environment

Are they indeed runnable in containers and this makes sense?

According to this documentation it is possible to run windows in docker. Yes it could make sense as this would be comparable to the positive effects of running linux in docker. Note that:

  Disclaimer: This lab is still in work, and is based off of the blog,
  but you can test and leverage the example walkthroughs now, if you
  want to start experimenting. Please check back as the lab evolves.

Whether it makes sense: From a Linux perspective it does not as a Windows container will be significantly larger than a Linux one at the moment. From a Windows perspective it does not make sense as well as Windows itself is not as lightweight at the moment as Linux. Imagine that one runs 50 war files on a dedicated Windows server at the moment. If one would use Windows Docker containers for that purpose then a dedicated server should be used as a base image, each war file should be run in a separate container. This means that the number of required resource would grow exponentially. I assume that the creators of the native Windows Docker container take this into account and that this issue will be solved in the future.

  Are there other Windows native containerization scenarios?

What is the definition of a microservice? One could also argue that running SQLServer and IIS are microservices if one compares it with running IIS and SQLServer and other Windows services on a Windows server instead of separately. If one compares it with running Linux in docker then one could use a windows server as a base image by defining FROM windows in a dockerfile and run tomcat in a Windows container instead of running it on a dedicated server.


Ansible service module running state utility

It's deprecated in favor of started.

TASK [Test postfix is running]
**************************************************************
[DEPRECATION WARNING]: state=running is deprecated. Please use
state=started. This feature will be removed in version 2.7.
Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.


How do I deploy an applications to Kubernetes with Minikube?

Welcome to the wonderful world of container orchestrators. It is considered bad practice to install many apps into a single container. The new way of working is not to install an application server that runs many WAR files. The new way of deploying is to have each app be a self-contained container where your build tool (eg maven) packaged the servlet container and your app together into a single file. Then each container can be deployed to Kubernetes and independently scaled up and restarted. They can also have memory leaks and crash without effecting each other.

One good example of that is springboot where in your pom.xml you can tell it to build an executable JAR. This will be large as it will include Tomcat into it. When you run the executable file it will launch java and Tomcat and bind to a port and run your app. You don't have to use springboot you can simply add the Tomcat or Jetty plugin to your maven project and have the container just launch "mvn jetty:run" or "mvn tomcat7:run".

I recommend you try minishift which is like minikube but the openshift distro of kubernetes that has more devops features. If you want to deploy Java Servlet apps you just use its built in catalog, select "Open JDK" supply it with your git repo, and it will checkout your code, build the app with maven, create a container image, and deploy the it, with a route and URL you can access it. Yet you can create things from scratch the catalog is just creating all the necessary YAML. Here is the sample code that you would use as your maven project that can be built and run by the "Open JDK" catalog. The  pom.xml shows that when the code is built with mvn -P openshift package it puts the war file into a deployments folder and jboss will run it when the container is started. If you don't like jboss then you can add a .s2i/bin/run script that starts your compiled application another way (e.g. "mvn jetty:run" or "mvn tomcat 7:run")

Helm is a package manager for Kubernetes it works with openshift extensions also. Helm tracks what your have installed and lets you rollback. The official chart repos have ready made charts to install databases and redis etc. I made a generic chart that runs any container on openshift kubernetes and we use it to install all our apps whether mode.js or php or whatever. It uses the same YAML as the built in templates use.

When you want to install many charts then you can use Helmfile. Your Helmfile has all the yaml to install all your helm charts. The best feature is that it only modifies what changed. If you modify one setting in a Helmfile that configures many apps it will only update the one app. And let you see the prior versions of the k8s yaml of the apps store in k8s and lets you easily rollback to the last working state.

I am making open source the stuff we run a business with as “OCD” where you change the config in git and a webook will run Helmfile to update your Kubernetes env. Here is the minishift demo how-to.


Update: Docker just released support for Kubernetes as scheduler, which changes the situation and makes Kubernetes just an alternative scheduler to Docker Swarm.

TL;DR: DON'T DO IT. Engineers always try to create these dog-pigs. Every unnecessary technology you bring will bring another whole set of faults. If you can pick one, then pick one and be happy you don't have to do both. If you like to play with Kubernetes, just get a private account on Google Cloud and play with it as much as you like. But don't make everyone at your company suffer through unnecessary complications.

Those are two parallel and mostly equivalent technologies. If your business had legitimate business reason to deploy in multiple Cloud providers for reliability for example and wanted to deploy into both AWS ECS (Elastic Container Service - based on Docker) and Google GKE (Container Engine - based on Kubernetes) and you were asking how do you build a pipeline, which would build your software and package in containers for deployment in both, that would be something else, but doing it just because you want to play with a new technology is very irresponsible.


initContainer with huge size software like magento2

Static assets should not be in the image, but rather on a PersistentVolume or better yet, on an object store such as AWS S3 then let a CDN to use it.

At the end of the day, Magento is a CMS, so the admin users will change the assets and other part of the content. It is not practical to bake assets these in to an container image doe the nature Magento.


Checking Jenkins build status from only one pipeline and avoid from other pipeline pointing to same gitlab repo in PR build

As far as I can tell from reading the source code, the build status notifications are basically hard-coded into the GitLab Branch Source plugin and cannot be disabled.  Therefore, without writing your own plugin or something like that, the only way I can think of to disable the build status notifications would be to use a different method of checking out the repository that does not use the GitLab plugin, e.g. using the checkout step with a direct url.


Jenkins view is getting duplicate every container restart

The solution was to remove the directory init.groovy.d from Jenkins home folder (inside the container)


Checkout this solution with codepipeline.  https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/


Jenkins SSH connection on remote server immediately closes after login

Check to see if the home directory for Jenkins was created?

/home/jenkins


Installing it as a tool allows Jenkins to manage the binary and ensure it's available for your build while also providing supporting pipeline syntax withEnv(). Having an agent with the binary allows you to shell out and run your commands using sh without withEnv(). It also allows you to decide where these binaries come from depending on your agent type (static, vm, container, pod).

Personally, I prefer the latter because I can control the updating of said binaries on the agent in a more automated fashion.